Dynamical Variational Autoencoders

class: middle, title-slide
count: false

.center[
# Dynamical Variational Autoencoders
 
 .bold[Simon Leglaive]
 CentraleSupélec, IETR

.medium[joint work with] 
 .medium[.bold[Laurent Girin] .tiny[1], $\,$ .bold[Xiaoyu Bie] .tiny[2], $\,$ .bold[Julien Diard] .tiny[3], $\,$ .bold[Thomas Hueber] .tiny[1], $\,$ .bold[Xavier Alameda-Pineda] .tiny[2], $\,$ .bold[Radu Horaud] .tiny[2]]
 
 .tiny[1 Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab $\qquad$ 2 Inria, Univ. Grenoble Alpes, CNRS, LJK $\qquad$ 
 
 3 Univ. Grenoble Alpes, CNRS, LPNC]

.tiny[SigMA seminar - CRIStAL, Lille, France January 18, 2021]
]

---
class: center, middle
count: false

# Introduction

---
## Supervised learning

A great number of successful practical applications of machine and deep learning rely on **supervised learning** methods.

.grid[

.kol-2-3[
<div style="text-align:center;margin-bottom:30px">
 <iframe width="650" height="350" src="https://www.youtube.com/embed/qWl9idsCuLQ?start=10&autoplay=1&mute=1" frameborder="0" allow="autoplay; encrypted-media" style="max-width:100%" allowfullscreen=""></iframe>
 </div>

.caption[ .tiny[Zhao et al., "[ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://arxiv.org/pdf/1704.08545.pdf)", ECCV 2018 ]]
  
]
.kol-1-3[
.tiny[
The Cityscapes dataset for semantic segmentation of urban images contains **5k images** with high quality pixel-level annotations.

"Annotation and quality control required more than **1.5 h on average for a single image**" (Cordts et al., 2016).

More than 300 days of annotation!
]
]
]

.credit[M. Cordts et al., ["The Cityscapes Dataset for Semantic Urban Scene Understanding"](https://www.cityscapes-dataset.com/wordpress/wp-content/papercite-data/pdf/cordts2016cityscapes.pdf), IEEE CVPR, 2016]

---
## Unsupervised learning

We need **unsupervised** methods that can learn to unveil the **underlying (hidden) structure** of the data without requiring ground-truth labels.

**Semi-supervised methods** are also of great interest.

.center.width-70[![](images/genesis.png)]

.tiny.caption[
Component-by-component scene generation with GENESIS, a generative model of 3D scenes capable of both decomposing and generating scenes by capturing relationships between scene components]

.credit[Image credits: Engelcke et al., "[GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://arxiv.org/pdf/1907.13052.pdf)", ICLR 2020.]

???

- GENESIS pursues a consistent strategy for scene generation: Step one generates the floor and the sky, defining the layout of the scene. Steps two to four generate individual foreground objects. Some of these slots remain empty if less than three objects are present in the scene. The final three steps generate the walls in the background.

- Unsupervised models can also be used for **semi-supervised learning**, where the goal is to exploit both a **small-scale labeled** dataset and a **large-scale unlabeled** dataset.

The unlabeled data are used to learn a representation that makes the supervised task simpler.

The resulting semi-supervised model performs typically better than a supervised model trained on the small labeled dataset.

---
## Generative modeling

.small-vspace.center.width-50[![](images/DGM.png)]

The goal is to tune the parameters $\theta$ of the model distribution $p(\mathbf{x} ; \theta)$ so that it is 
as close as possible to the true data distribution $p^\star(\mathbf{x})$, according to some measure of fit.

For instance, minimizing the Kullback-Leibler (KL) divergence is equivalent to maximum likelihood estimation of $\theta$.

---
## 1-dimensional example

We assume a simple Gaussian generative model of data $x \in \mathbb{R}$:

$$ p(x ; \theta) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left( \frac{(x - \mu)^2 }{ 2 \sigma^2} \right), $$

where the parameters $\theta = \\{ \mu, \sigma^2 \\}$ are estimated in the maximum likelihood sense.

.center.width-50[![](images/illust_Gaussian.svg)]

---
## $10^6$-dimensional example

.vspace[

]

.credit[T. Karras et al., [Analyzing and Improving the Image Quality of StyleGAN](https://arxiv.org/abs/1912.04958), CVPR 2020.]

---
## Deep generative models (DGM)

- The model distribution $p(\mathbf{x} ; \theta)$ is somehow defined by means of a neural network.

- Two seminal models: Variational autoencoders (VAEs) .tiny[(Kingma and Welling, 2014; Rezende et al., 2014)] and generative adversarial networks (GANs) .tiny[(Goodfellow et al., 2014)].

- DGM contain **hundreds of thoughsands of parameters**, trained in a scalable way using **large datasets of unlabeled high-dimensional data**.

.center.width-60[![](images/GAN_progress.jpeg)]

.credit[

I. Goodfellow et al., "[Generative Adversarial Networks](https://arxiv.org/abs/1406.2661)", NeurIPS 2014.

D.P. Kingma and M. Welling, "[Auto-Encoding Variational Bayes](https://arxiv.org/pdf/1312.6114.pdf)", ICLR 2014.

D.J. Rezende et. al, "[ Stochastic backpropagation and approximate inference in deep generative models](https://arxiv.org/pdf/1401.4082.pdf)", ICML 2014.
  
]

---
## DGMs with latent variables

The model distribution is defined as a marginal distribution, by introducing a low-dimensional latent random vector:

$$ p(\mathbf{x} ; \theta) = \int p(\mathbf{x}, \mathbf{z} ; \theta) d\mathbf{z} = \int p(\mathbf{x} | \mathbf{z} ; \theta) p(\mathbf{z}) d\mathbf{z}. $$

.center.width-50[![](images/DGM_latent_2.png)]

GANs and VAEs are two examples of latent-variable-based DGM, but $p(\mathbf{x} | \mathbf{z} ; \theta)$ is only defined explicitely (i.e. analytically) for VAEs.

---
## Sequential data

Most of the focus of deep generative modeling has been on static data (e.g. images).

But many data are **sequential/temporal** by nature (e.g. videos, music, speech, text).

We need models that can learn the **temporal dependencies** between the data (observed and latent) at different time frames.

.small-vspace.center[
<img src="images/Muybridge_race_horse_animated.gif", width=370>
]
.caption[
.tiny[
["The horse in motion"](https://en.wikipedia.org/wiki/The_Horse_in_Motion), pictures made by Eadweard Muybridge in 1887. 
]
]

.small-vspace[

]

---
# Today's presentation

.bold[Objective]:

Introduce and discuss a class of models called **Dynamical Variational
Autoencoders** (DVAEs) that encompass a large subset of temporal extensions of VAE that have been proposed in the literature.

.bold[Outline]:

1. Variational autoencoders
2. Dynamical variational autoencoders
3. Application in speech enhancement

.footnote[This presentation is mostly based on:

L. Girin, S. L., X. Bie, J. Diard, T. Hueber, X. Alameda-Pineda, ["Dynamical Variational Autoencoders: A Comprehensive Review"](https://arxiv.org/pdf/2008.12595.pdf), arXiv preprint arXiv:2008.12595, 2020.

S. L., X. Alameda-Pineda, L. Girin, R. Horaud, ["A Recurrent Variational Autoencoder for Speech Enhancement"](https://arxiv.org/pdf/1910.10942.pdf), IEEE ICASSP, 2020.
]

---
class: middle
count: false

.center[
  # Variational autoencoders 
]

.big-vspace[

]

- .bold[Generative model]
- Inference model and training
- Application examples

.credit[

D.P. Kingma and M. Welling, [Auto-Encoding Variational Bayes](https://arxiv.org/pdf/1312.6114.pdf), ICLR 2014.

D.J. Rezende et. al, [ Stochastic backpropagation and approximate inference in deep generative models](https://arxiv.org/pdf/1401.4082.pdf), ICML 2014.
  
]

---
## Generative model

Let $\mathbf{x} \in \mathbb{R}^D$ and $\mathbf{z} \in \mathbb{R}^K$ be two random vectors (typically $K \ll D$). The generative model is defined by:

$$ p(\mathbf{x} ; \theta) = \int p(\mathbf{x} | \mathbf{z} ; \theta) p(\mathbf{z}) d\mathbf{z}.$$

.grid[
.kol-2-3[

- The **prior** is a standard Gaussian distribution:

$$ p(\mathbf{z}) = \mathcal{N}(\mathbf{z}; \mathbf{0}, \mathbf{I}).$$

- The **likelihood** is parametrized with a **generative/decoder neural network**, e.g.

$$ p(\mathbf{x} | \mathbf{z} ; \theta ) = \mathcal{N}\left( \mathbf{x}; \boldsymbol{\mu}\_\theta(\mathbf{z}), \text{diag}\left\\{ \mathbf{v}\_\theta(\mathbf{z}) \right\\} \right), $$

where $\theta$ denotes the parameters of the decoder network.

]

.kol-1-3[

.right.width-80[![](images/VAE_decoder.png)]

]
]

---
## Parameters estimation

- Direct maximum marginal likelihood estimation is intractable due to non-linearities.

- For any distribution $q(\mathbf{z} | \mathbf{x}; \phi)$, we have .tiny[(Neal and Hinton, 1999; Jordan et al. 1999)]

$$ \ln p(\mathbf{x}; \theta) = \mathcal{L}(\mathbf{x}; \phi, \theta) + D\_{\text{KL}}(q(\mathbf{z} | \mathbf{x}; \phi) \parallel p(\mathbf{z} | \mathbf{x}; \theta)),$$

where $\mathcal{L}(\mathbf{x}; \phi, \theta)$ is the **evidence lower bound** (ELBO), defined by

$$ \mathcal{L}(\mathbf{x}; \phi, \theta) = \mathbb{E}\_{q(\mathbf{z} | \mathbf{x}; \phi)} [\ln p(\mathbf{x}, \mathbf{z}; \theta) - \ln q(\mathbf{z} | \mathbf{x}; \phi)]. $$

.credit[
  
  R.M. Neal and G.E. Hinton, ["A view of the EM algorithm that justifies incremental, sparse, and other variants"](http://www.cs.toronto.edu/~radford/ftp/emk.pdf), in M. I. Jordan (Ed.), .italic[Learning in graphical models], Cambridge, MA: MIT Press, 1999.

M.I. Jordan et al., ["An introduction to variational methods for graphical models"](https://people.eecs.berkeley.edu/~jordan/papers/variational-intro.pdf),  Machine learning, 1999.]

--
count: false

.left-column.center[

.bold[Problem #1]

$$ \underset{\theta}{\max}\, \mathcal{L}(\mathbf{x}; \phi, \theta),$$
where $\mathcal{L}(\mathbf{x}; \phi, \theta) \le \ln p(\mathbf{x}; \theta)$

]

--
count: false

.right-column.center[

.bold[Problem #2]
$$ \underset{\phi}{\max}\, \mathcal{L}(\mathbf{x}; \phi, \theta) $$
$$ \Leftrightarrow  \underset{\phi}{\min}\, D\_{\text{KL}}(q(\mathbf{z} | \mathbf{x}; \phi) \parallel p(\mathbf{z} | \mathbf{x}; \theta))$$
]

.reset-column[
]

---
class: center, middle

To fully define the objective function, we need to specify the **inference model** $q(\mathbf{z} | \mathbf{x}; \phi)$.

---
class: middle
count: false

.center[
  # Variational autoencoders
]

.big-vspace[

]

- Generative model
- .bold[Inference model and training]
- Application examples

.credit[

D.P. Kingma and M. Welling, [Auto-Encoding Variational Bayes](https://arxiv.org/pdf/1312.6114.pdf), ICLR 2014.

D.J. Rezende et. al, [ Stochastic backpropagation and approximate inference in deep generative models](https://arxiv.org/pdf/1401.4082.pdf), ICML 2014.
  
]

---
## Inference model

The **inference model** (approximate posterior) is typically defined by:

$$ q(\mathbf{z} | \mathbf{x}; \phi) =  \mathcal{N}\left( \mathbf{z}; \boldsymbol{\mu}\_\phi(\mathbf{x}), \text{diag}\left\\{ \mathbf{v}\_\phi(\mathbf{x}) \right\\} \right),$$

where the mean and variance vectors are provided by the **encoder** neural network.

.vspace.center.width-30[![](images/VAE_encoder.png)]

---
## ELBO

The ELBO is now fully defined:

$$
  \begin{aligned}
  \mathcal{L}(\mathbf{x}; \phi, \theta) &= \mathbb{E}\_{q(\mathbf{z} | \mathbf{x}; \phi)} [\ln p(\mathbf{x}, \mathbf{z}; \theta) - \ln q(\mathbf{z} | \mathbf{x}; \phi)] \\\\
  &= \underbrace{\mathbb{E}\_{q(\mathbf{z} | \mathbf{x}; \phi)} [\ln p(\mathbf{x} | \mathbf{z}; \theta)]}\_{\text{reconstruction accuracy}} - \underbrace{D\_{\text{KL}}(q(\mathbf{z} | \mathbf{x}; \phi) \parallel p(\mathbf{z}))}\_{\text{regularization}}.
  \end{aligned}
$$
- prior: $ \hspace{2cm} p(\mathbf{z}) = \mathcal{N}(\mathbf{z}; \mathbf{0}, \mathbf{I})$
- likelihood model: $ \hspace{.45cm} p(\mathbf{x} | \mathbf{z} ; \theta ) = \mathcal{N}\left( \mathbf{x}; \boldsymbol{\mu}\_\theta(\mathbf{z}), \text{diag}\left\\{ \mathbf{v}\_\theta(\mathbf{z}) \right\\} \right)$
- inference model: $ \hspace{.42cm} q(\mathbf{z} | \mathbf{x}; \phi) =  \mathcal{N}\left( \mathbf{z}; \boldsymbol{\mu}\_\phi(\mathbf{x}), \text{diag}\left\\{ \mathbf{v}\_\phi(\mathbf{x}) \right\\} \right)$

--
count: false

The reconstruction accuracy term is approximated with a Monte Carlo estimate:

---
## ELBO

The ELBO is now fully defined:

The reconstruction accuracy term is approximated with a Monte Carlo estimate, using the so-called **reparametrization trick**:

$$\mathbb{E}\_{q(\mathbf{z} | \mathbf{x}; \phi)} [\ln p(\mathbf{x} | \mathbf{z}; \theta)] \approx \frac{1}{R} \sum\_{r=1}^R \ln p(\mathbf{x} | \tilde{\mathbf{z}}\_r ; \theta), \qquad \begin{cases} \boldsymbol{\epsilon}\_r &\sim \mathcal{N}(\mathbf{0}, \mathbf{I})  \\\\ \tilde{\mathbf{z}}\_r &= \boldsymbol{\mu}\_\phi(\mathbf{x}) + \text{diag}\left\\{ \mathbf{v}\_\phi(\mathbf{x}) \right\\}^{\frac{1}{2}} \boldsymbol{\epsilon}\_r \end{cases}. $$

---
## Training procedure 
** Step 1: Pick an example in the dataset **

$$
  \begin{aligned}
  \mathcal{L}(\mathbf{x}; \phi, \theta) = \ln p({\color{brown}\mathbf{x}} | \tilde{\mathbf{z}}; \theta) - D\_{\text{KL}}(q(\mathbf{z} | {\color{brown}\mathbf{x}}; \phi) \parallel {\color{green}p(\mathbf{z})})
  \end{aligned}
$$

.center.width-80[![](images/training_1.png)]

---
## Training procedure 
** Step 2: Map through the encoder **

$$
  \begin{aligned}
  \mathcal{L}(\mathbf{x}; \phi, \theta) = \ln p({\color{green}\mathbf{x}} | \tilde{\mathbf{z}}; \theta) - D\_{\text{KL}}({\color{brown} q(\mathbf{z} | } {\color{green}\mathbf{x} } {\color{brown}; \phi)} \parallel {\color{green}p(\mathbf{z})})
  \end{aligned}
$$

.center.width-80[![](images/training_2.png)]

---
## Training procedure 
** Step 3: Sample from the inference model **

$$
  \begin{aligned}
  \mathcal{L}(\mathbf{x}; \phi, \theta) = \ln p({\color{green}\mathbf{x}} | {\color{brown}\tilde{\mathbf{z}}}; \theta) -{\color{green}  D\_{\text{KL}}(q(\mathbf{z} | \mathbf{x}; \phi) \parallel p(\mathbf{z}))}
  \end{aligned}
$$

.center.width-80[![](images/training_3.png)]

---
## Training procedure 
** Step 4: Map through the decoder **

$$
  \begin{aligned}
  \mathcal{L}(\mathbf{x}; \phi, \theta) = {\color{green}\ln p(\mathbf{x} | \tilde{\mathbf{z}}; \theta)} -{\color{green}  D\_{\text{KL}}(q(\mathbf{z} | \mathbf{x}; \phi) \parallel p(\mathbf{z}))}
  \end{aligned}
$$

.center.width-80[![](images/training_4.png)]

---
## Training procedure 
** Step 5: Backpropagate and gradient step **

$$
  \begin{aligned}
  \mathcal{L}(\mathbf{x}; \phi, \theta) = \ln p(\mathbf{x} | \tilde{\mathbf{z}}; \theta) - D\_{\text{KL}}(q(\mathbf{z} | \mathbf{x}; \phi) \parallel p(\mathbf{z}))
  \end{aligned}
$$

.center.width-80[![](images/training_5.png)]

Encoder-decoder shape, which correspond to an inference-generation process.

.footnote[You can also average over mini batches before doing the backpropagation.]

---
## At test time

.center.width-100[![](images/testing.png)]

- Note that the encoder was only introduced in order to estimate the parameters of the decoder.

- We do not need the encoder for generating new samples.

- But it is useful if we need to do inference.

---
class: middle
count: false

.center[
  # Variational autoencoders
]

.big-vspace[

]

- Generative model
- Inference model and training
- .bold[Application examples]

---
## Generated MNIST samples

.vspace.center.width-40[![](images/generated_samples_MNIST_K=16.png)]

.caption[VAE with $K=16$]

---
class: middle, center

.center[<img src="./images/walking_latent_space.png" style="width: 450px;" />]

.tiny[Linearly spaced coordinates on the unit square are transformed through the inverse cumulative distribution function of the Gaussian to produce
values of the latent variables which are then mapped through the decoder network.]

---
## Encoding-decoding of speech with a VAE

.grid[
.kol-1-3[

original signal

reconstructed signal

.vspace.tiny[
$\mathbf{x}$ is an STFT frame of dimension $513$.

$\mathbf{z}$ is a latent vecotr of dimension $16$.

Model trained on Wall Street Journal dataset (25 hours).
  ]

]
.kol-2-3[
  .center.width-80[![](images/reconstruction_VAE.svg)]
]

]

.credit[Audio file from voiceHome-2 corpus, N. Bertin, E. Camberlein, R. Lebarbenchon, E. Vincent and F. Bimbot, [URL](http://voice-home.gforge.inria.fr/voiceHome-2_corpus.html)]

---
## Generation of speech with a VAE

.grid[
.vspace.kol-1-3[

]
.kol-2-3[

.center.width-90[![](images/generated_VAE.svg)]

]

The lack of temporal modeling is clearly a problem.

.footnote[Phase estimated with Griffin-Lim algorithm.]

---
## Encoding-decoding of speech with a **dynamical** VAE

.grid[
.kol-1-3[

original signal

reconstructed signal

<audio controls src="audio/signal_recon_SRNN_oracle_phase_latent_dim=16.wav"></audio>
]
.kol-2-3[
 .center.width-90[![](images/reconstruction_SRNN.svg)]
]

]

.credit[Audio file from voiceHome-2 corpus, N. Bertin, E. Camberlein, R. Lebarbenchon, E. Vincent and F. Bimbot, [URL](http://voice-home.gforge.inria.fr/voiceHome-2_corpus.html)]

---
## Generation of speech with a **dynamical** VAE

.grid[
.vspace.kol-1-3[

- Phoneme structure
- Voiced/unvoiced phonemes
- Coarticulation
- Silences

]
.kol-2-3[

.center.width-90[![](images/generated_SRNN.svg)]

]

.footnote[Phase generated with Griffin-Lim algorithm.]

---
class: middle
count: false

.center[
  # Dynamical VAEs
]

.vspace[

]
- .bold[Generative model]
- Inference model and training

---
class: middle

We are now interested in modeling **sequential data**:

- **Observed** sequence $\mathbf{x}\_{1:T} = \\{\mathbf{x}\_t \in \mathbb{R}^D\\}_{t=1}^T$
- **Latent** sequence $\mathbf{z}\_{1:T} = \\{\mathbf{z}\_t \in \mathbb{R}^L\\}_{t=1}^T$

Generative modeling consists in defining the joint distribution:

$$p\_\theta(\mathbf{x}\_{1:T}, \mathbf{z}\_{1:T}).$$

.vspace[

]

.footnote[In the following, to simplify the presentation we may omit to denote the model parameters when it is not confusing.]

---
## Chain rule