+ - 0:00:00
Notes for current slide
Notes for next slide

\global\def\myx#1{{\color{green}\mathbf{x}_{#1}}} \global\def\myxa#1{{\color{green}\mathbf{x}_{#1}^{(a)}}} \global\def\myza#1{{\color{green}\mathbf{z}_{#1}^{(a)}}} \global\def\myxv#1{{\color{purple}\mathbf{x}_{#1}^{(v)}}} \global\def\myzv#1{{\color{purple}\mathbf{z}_{#1}^{(v)}}} \global\def\myzav#1{{\color{brown}\mathbf{z}_{#1}^{(av)}}} \global\def\myzds#1{{\color{brown}\mathbf{z}_{#1}^{(av)}}} \global\def\mywav{{\color{brown}\mathbf{w}^{(av)}}} \global\def\myw{{\color{brown}\mathbf{w}}} \global\def\mys#1{{\color{green}\mathbf{s}_{#1}}} \global\def\myS#1{{\color{green}\mathbf{S}_{#1}}} \global\def\myz#1{{\color{brown}\mathbf{z}_{#1}}} \global\def\myztilde#1{{\color{brown}\tilde{\mathbf{z}}_{#1}}} \global\def\myhnmf#1{{\color{brown}\mathbf{h}_{#1}}} \global\def\myztilde#1{{\color{brown}\tilde{\mathbf{z}}_{#1}}} \global\def\myu#1{\mathbf{u}_{#1}} \global\def\mya#1{\mathbf{a}_{#1}} \global\def\myv#1{\mathbf{v}_{#1}} \global\def\mythetaz{\theta_\myz{}} \global\def\mythetax{\theta_\myx{}} \global\def\mythetas{\theta_\mys{}} \global\def\mythetaa{\theta_\mya{}} \global\def\bs#1{{\boldsymbol{#1}}} \global\def\diag{\text{diag}} \global\def\mbf{\mathbf} \global\def\myh#1{{\color{purple}\mbf{h}_{#1}}} \global\def\myhfw#1{{\color{purple}\overrightarrow{\mbf{h}}_{#1}}} \global\def\myhbw#1{{\color{purple}\overleftarrow{\mbf{h}}_{#1}}} \global\def\myg#1{{\color{purple}\mbf{g}_{#1}}} \global\def\mygfw#1{{\color{purple}\overrightarrow{\mbf{g}}_{#1}}} \global\def\mygbw#1{{\color{purple}\overleftarrow{\mbf{g}}_{#1}}} \global\def\neq{\mathrel{\char`≠}}

A Multimodal Dynamical Variational Autoencoder for Audiovisual Speech Representation Learning

Simon Leglaive

CentraleSupélec, IETR (UMR CNRS 6164), France

Seminar @ LISTEN joint laboratory

Palaiseau (France) - January 12, 2023

1

Joint work with


Samir Sadok1


Laurent Girin2


Xavier Alameda-Pineda3


Renaud Séguier1

1 CentraleSupélec, IETR (UMR CNRS 6164), France

2 Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France

3 Inria, Univ. Grenoble Alpes, CNRS, LJK, France

2

🚧 \hspace{.5cm} Ongoing work \hspace{.5cm} 🚧

3

Introduction

3

Supervised learning

The majority of successful applications of machine/deep learning rely on supervised learning.

Semantic urban image segmentation with ICNet (Zhao et al., 2018)

The Cityscapes dataset
(Cordts et al., 2016)

  • 5k images with high quality pixel-level annotations
  • 1.5h to annotate each single image

More than 300 days of annotation! 😱

It is intractable to collect labels for every scenario and task.


M. Cordts et al., The Cityscapes dataset for semantic urban scene understanding, IEEE CVPR 2016
Zhao et al., ICNet for real-time semantic segmentation on high-resolution images, ECCV 2018

4

Unsupervised learning

We need unsupervised methods that can learn to unveil the underlying structure of the data without or with few ground-truth labels.

GENESIS (Engelcke et al., 2020), a generative model of 3D scenes capable of both decomposing and generating scenes by capturing relationships between scene components. Image credits: (Engelcke et al., 2020).

Deep latent variable generative models have emerged as promising approaches.

M. Engelcke et al., GENESIS: Generative scene inference and sampling with object-centric latent representations, ICLR 2020.

5

Generative modeling of structured high-dimensional data

High-dimensional data x∈Rd\myx{} \in \mathbb{R}^d such as natural images or speech signals exhibit some form of regularity, preventing their dimensions from varying independently.


From a generative perspective, this regularity suggests that there exists a smaller dimensional latent variable z∈Rℓ\myz{} \in \mathbb{R}^\ell that generated x∈Rd\myx{} \in \mathbb{R}^d, ℓ≪d\ell \ll d.

Picture credits: wayhomestudio on Freepik.

6

Latent-variable generative modeling

  • Generative modeling consists in estimating the parameters θ\theta so that pθ(x)≈p⋆(x)p_\theta(\myx{}) \approx p^\star(\myx{}) according to some measure of fit, for instance the Kullback-Leibler (KL) divergence.

  • When the model includes a deep neural network, we obtain a deep generative model.

7

The variational autoencoder (VAE) (Kingma and Welling, 2014; Rezende et. al., 2014)

Prior

p(z)=N(z;0,I) \small p(\myz{}) = \mathcal{N}(\myz{}; \mathbf{0}, \mathbf{I})

Generative model

pθ(x∣z)=N(x;μθ(z),Σθ(z)) \small p_\theta(\myx{} | \myz{}) = \mathcal{N}\left( \myx{}; \boldsymbol{\mu}_\theta(\myz{}), \boldsymbol{\Sigma}_\theta(\myz{}) \right)

Inference model

qϕ(z∣x)=N(z;μϕ(x),Σϕ(x))\small q_\phi(\myz{} | \myx{}) = \mathcal{N}\left( \myz{}; \boldsymbol{\mu}_\phi(\myx{}), \boldsymbol{\Sigma}_\phi(\myx{}) \right) \\


The inference model approximates the intractable exact posterior distribution

pθ(z∣x)=pθ(x∣z)p(z)∫pθ(x∣z)p(z)dz\displaystyle p_\theta(\myz{} | \myx{}) = \frac{p_\theta(\myx{} | \myz{})p(\myz{})}{\int p_\theta(\myx{} | \myz{})p(\myz{})d\myz{}}

Σϕ(x)=diag{vϕ(x)}Σθ(z)=diag{vθ(z)}\footnotesize \hspace{.5cm} \boldsymbol{\Sigma}_\phi(\myx{}) = \diag\{ \mathbf{v}_\phi(\myx{})\} \qquad\qquad \boldsymbol{\Sigma}_\theta(\myz{}) = \diag\{ \mathbf{v}_\theta(\myz{}) \}

D.P. Kingma and M. Welling, Auto-encoding variational Bayes, ICLR 2014.
D.J. Rezende et al., Stochastic backpropagation and approximate inference in deep generative models, ICML 2014.

8

The VAE parameters are estimated by maximizing the evidence lower bound (ELBO) (Neal and Hinton, 1999; Jordan et al. 1999) defined by: L(ϕ,θ)=Eqϕ(z∣x)[ln⁡pθ(x∣z)]⏟reconstruction accuracy−DKL(qϕ(z∣x)∥p(z))⏟regularization.\begin{aligned} \mathcal{L}(\phi, \theta) &= \underbrace{\mathbb{E}_{q_\phi(\myz{} | \myx{})} [\ln p_\theta(\myx{} | \myz{})]}_{\text{reconstruction accuracy}} - \underbrace{D_{\text{KL}}(q_\phi(\myz{} | \myx{}) \parallel p(\myz{}))}_{\text{regularization}}. \end{aligned}

The ELBO can also be decomposed as: L(ϕ,θ)=ln⁡pθ(x)−DKL(qϕ(z∣x)∥pθ(z∣x)).\begin{aligned} \mathcal{L}(\phi, \theta) &= \ln p_\theta(\myx{}) - D_{\text{KL}}(q_\phi(\myz{} | \myx{}) \parallel p_\theta(\myz{} | \myx{})). \end{aligned}

Generative model parameters estimation

max⁡θ {L(ϕ,θ)≤ln⁡pθ(x)} \underset{\theta}{\max}\, \Big\{ \mathcal{L}(\phi, \theta) \le \ln p_\theta(\myx{}) \Big\}

Inference model parameters estimation

max⁡ϕ L(ϕ,θ)  ⇔  min⁡ϕ DKL(qϕ(z∣x)∥pθ(z∣x)) \underset{\phi}{\max}\, \mathcal{L}(\phi, \theta) \,\,\Leftrightarrow\,\, \underset{\phi}{\min}\, D_{\text{KL}}(q_\phi(\myz{} | \myx{}) \parallel p_\theta(\myz{} | \myx{}))

R.M. Neal and G.E. Hinton, A view of the EM algorithm that justifies incremental, sparse, and other variants, in M. I. Jordan (Ed.), Learning in graphical models, 1999.
M.I. Jordan et al., An introduction to variational methods for graphical models, Machine Learning, 1999.

9

  • A trained VAE can be used for generation, transformation, and downstream tasks.

  • Ideally, the learned representation should be disentangled (Higgins et al., 2018), i.e., somehow easy to relate to independent and interpretable high-level characteristics of the data.

Supervised learning from disentangled representations has been found to be more sample-efficient, more robust, and better in terms of generalization (van Steenkiste et al., 2019).

I. Higgins et al., Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230, 2018.
S. Sadok et al., Learning and controlling the source-filter representation of speech with a variational autoencoder, arXiv preprint arXiv:2204.07075, 2022.
S. van Steenkiste et al., Are disentangled representations helpful for abstract visual reasoning?, NeurIPS, 2019.
10

Over the past few years, the VAE has been extended in many ways, including for processing dynamical or multimodal data.

In this talk, we will present a multimodal and dynamical VAE (MDVAE) applied to unsupervised audiovisual speech representation learning.

L. Girin et al., Dynamical variational autoencoders: A comprehensive review, Foundations and Trends in Machine Learning, 2021.
M. Suzuki et al., Joint multimodal learning with deep denerative models, ICLR Workshop, 2017.
M. Wu and N. Goodman, Multimodal generative models for scalable weakly-supervised learning, NeurIPS, 2018.
W.-N. Hsu and J. R. Glass, Disentangling by partitioning: A representation learning framework for multimodal sensory data, arXiv preprint arXiv:1805.11264, 2018.
Y. Shi et al., Variational mixture-of-experts autoencoders for multi-modal deep denerative models, NeurIPS, 2019.
T. Sutter et al., Multimodal generative learning utilizing Jensen-Shannon divergence, NeurIPS, 2020.
T. Sutter et al., Generalized Multimodal ELBO, ICLR, 2021.
11

Audiovisual (AV) speech latent factors

  • Speaker's identity and global emotional state
    ↪ static, shared (AV)

  • Lip movements, phonemic information (part of)
    ↪ dynamical, shared (AV)

  • Other facial movements and head pose
    ↪ dynamical, modality-specific (V)

  • Pitch variations, phonemic information (part of)
    ↪ dynamical, modality-specific (A)

We seek to learn a multimodal dynamical VAE that disentangles these AV speech latent factors: dynamical and modality-specific, dynamical and audiovisual, static and audiovisual.

Video credits: K. Wang et al., MEAD: A large-scale audio-visual dataset for emotional talking-face generation, ECCV, 2020.

12

Multimodal dynamical VAE (MDVAE)


  • Generative model
  • Inference model
  • Two-stage training
12

Notations

x(a)∈Rda×T\myxa{} \in \mathbb{R}^{d_a \times T}

Observed dynamical audio data

x(v)∈Rdv×T\myxv{} \in \mathbb{R}^{d_v \times T}

Observed dynamical visual data

w∈Rℓw\myw \in \mathbb{R}^{\ell_w}

Latent static audiovisual data

z(av)∈Rℓav×T\myzav{} \in \mathbb{R}^{\ell_{av} \times T}

Latent dynamical audiovisual data

z(a)∈Rℓa×T\myza{} \in \mathbb{R}^{\ell_a \times T}

Latent dynamical audio
data

z(v)∈Rℓv×T\myzv{} \in \mathbb{R}^{\ell_v \times T}

Latent dynamical visual data

13

MDVAE generative model

  • Defining the generative model amounts to defining the joint distribution of all variables: pθ(x(a),x(v),z(av),z(a),z(v),w). p_\theta\left(\myxa{}, \myxv{}, \myzav{},\myza{},\myzv{}, \myw{}\right).

    By structuring the dependencies between these variables we hope to learn the desired disentangled representation of audiovisual speech in an unsupervised manner.

  • The temporal model in MDVAE is largely inspired by the disentangled sequential autoencoder (DSAE) of Li and Mandt (2018).

    MDVAE can be seen as a multimodal extension of DSAE.

Y. Li and S. Mandt, "Disentangled sequential autoencoder", ICML 2018.

14

The global probabilistic graphical model of MDVAE is defined by the following Bayesian network.

pθ(x(a),x(v),z(av),z(a),z(v),w)=pθ(x(a)∣z(av),z(a),w)pθ(x(v)∣z(av),z(v),w)pθ(z(av))pθ(z(a))pθ(z(v))pθ(w) \hspace{-.75cm} \small p_\theta\left(\myxa{}, \myxv{}, \myzav{},\myza{},\myzv{}, \myw{}\right) = p_\theta\left(\myxa{} \mid \myzav{},\myza{},\myw{}\right) p_\theta\left(\myxv{} \mid \myzav{}, \myzv{}, \myw{}\right) p_\theta\left(\myzav{}\right)p_\theta\left(\myza{}\right) p_\theta\left(\myzv{}\right)p_\theta\left(\myw{}\right)

15

The global probabilistic graphical model of MDVAE is defined by the following Bayesian network.

pθ(x(a),x(v),z(av),z(a),z(v),w)=pθ(x(a)∣z(av),z(a),w)pθ(x(v)∣z(av),z(v),w)pθ(z(av))pθ(z(a))pθ(z(v))pθ(w) \hspace{-.75cm} \small p_\theta\left(\myxa{}, \myxv{}, \myzav{},\myza{},\myzv{}, \myw{}\right) = \boxed{p_\theta\left(\myxa{} \mid \myzav{},\myza{},\myw{}\right)}p_\theta\left(\myxv{} \mid \myzav{}, \myzv{}, \myw{}\right) p_\theta\left(\myzav{}\right)p_\theta\left(\myza{}\right) p_\theta\left(\myzv{}\right)p_\theta\left(\myw{}\right)

15

The global probabilistic graphical model of MDVAE is defined by the following Bayesian network.

pθ(x(a),x(v),z(av),z(a),z(v),w)=pθ(x(a)∣z(av),z(a),w)pθ(x(v)∣z(av),z(v),w)pθ(z(av))pθ(z(a))pθ(z(v))pθ(w) \hspace{-.75cm} \small p_\theta\left(\myxa{}, \myxv{}, \myzav{},\myza{},\myzv{}, \myw{}\right) = p_\theta\left(\myxa{} \mid \myzav{},\myza{},\myw{}\right)\boxed{p_\theta\left(\myxv{} \mid \myzav{}, \myzv{}, \myw{}\right)} p_\theta\left(\myzav{}\right)p_\theta\left(\myza{}\right) p_\theta\left(\myzv{}\right)p_\theta\left(\myw{}\right)

15

The global probabilistic graphical model of MDVAE is defined by the following Bayesian network.

pθ(x(a),x(v),z(av),z(a),z(v),w)=pθ(x(a)∣z(av),z(a),w)pθ(x(v)∣z(av),z(v),w)pθ(z(av))pθ(z(a))pθ(z(v))pθ(w) \hspace{-.75cm} \small p_\theta\left(\myxa{}, \myxv{}, \myzav{},\myza{},\myzv{}, \myw{}\right) = p_\theta\left(\myxa{} \mid \myzav{},\myza{},\myw{}\right)p_\theta\left(\myxv{} \mid \myzav{}, \myzv{}, \myw{}\right) \boxed{p_\theta\left(\myzav{}\right)p_\theta\left(\myza{}\right) p_\theta\left(\myzv{}\right)p_\theta\left(\myw{}\right)}

15

The global probabilistic graphical model of MDVAE is defined by the following Bayesian network.

pθ(x(a),x(v),z(av),z(a),z(v),w)=pθ(x(a)∣z(av),z(a),w)pθ(x(v)∣z(av),z(v),w)pθ(z(av))pθ(z(a))pθ(z(v))pθ(w) \hspace{-.75cm} \small p_\theta\left(\myxa{}, \myxv{}, \myzav{},\myza{},\myzv{}, \myw{}\right) = p_\theta\left(\myxa{} \mid \myzav{},\myza{},\myw{}\right) p_\theta\left(\myxv{} \mid \myzav{}, \myzv{}, \myw{}\right) p_\theta\left(\myzav{}\right)p_\theta\left(\myza{}\right) p_\theta\left(\myzv{}\right)p_\theta\left(\myw{}\right)

To complete the generative model, we also need to define the temporal dependencies for the sequential variables.

15

pθ(x(a)∣z(av),z(a),w)=∏t=1Tpθ(xt(a)∣zt(av),zt(a),w),pθ(x(v)∣z(av),z(v),w)=∏t=1Tpθ(xt(v)∣zt(av),zt(v),w) \hspace{-.5cm} \small {p_\theta\left(\myxa{} \mid \myzav{},\myza{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxa{t} \mid \myzav{t},\myza{t},\myw{}\right)}, \hspace{.3cm} p_\theta\left(\myxv{} \mid \myzav{},\myzv{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxv{t} \mid \myzav{t},\myzv{t},\myw{}\right)

pθ(z(a))=∏t=1Tpθ(zt(a)∣z1:t−1(a)),pθ(z(av))=∏t=1Tpθ(zt(av)∣z1:t−1(av)),pθ(z(v))=∏t=1Tpθ(zt(v)∣z1:t−1(v)) \hspace{-.5cm} \small p_\theta\left(\myza{}\right) = \prod_{t=1}^T p_\theta\left(\myza{t} \mid \myza{1:t-1} \right), \hspace{.3cm} p_\theta\left(\myzav{}\right) = \prod_{t=1}^T p_\theta\left(\myzav{t} \mid \myzav{1:t-1} \right), \hspace{.3cm} p_\theta\left(\myzv{}\right) = \prod_{t=1}^T p_\theta\left(\myzv{t} \mid \myzv{1:t-1} \right)

16

pθ(x(a)∣z(av),z(a),w)=∏t=1Tpθ(xt(a)∣zt(av),zt(a),w),pθ(x(v)∣z(av),z(v),w)=∏t=1Tpθ(xt(v)∣zt(av),zt(v),w) \hspace{-.5cm} \small \boxed{p_\theta\left(\myxa{} \mid \myzav{},\myza{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxa{t} \mid \myzav{t},\myza{t},\myw{}\right)}, \hspace{.3cm} p_\theta\left(\myxv{} \mid \myzav{},\myzv{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxv{t} \mid \myzav{t},\myzv{t},\myw{}\right)

pθ(z(a))=∏t=1Tpθ(zt(a)∣z1:t−1(a)),pθ(z(av))=∏t=1Tpθ(zt(av)∣z1:t−1(av)),pθ(z(v))=∏t=1Tpθ(zt(v)∣z1:t−1(v)) \hspace{-.5cm} \small p_\theta\left(\myza{}\right) = \prod_{t=1}^T p_\theta\left(\myza{t} \mid \myza{1:t-1} \right), \hspace{.3cm} p_\theta\left(\myzav{}\right) = \prod_{t=1}^T p_\theta\left(\myzav{t} \mid \myzav{1:t-1} \right), \hspace{.3cm} p_\theta\left(\myzv{}\right) = \prod_{t=1}^T p_\theta\left(\myzv{t} \mid \myzv{1:t-1} \right)

16

pθ(x(a)∣z(av),z(a),w)=∏t=1Tpθ(xt(a)∣zt(av),zt(a),w),pθ(x(v)∣z(av),z(v),w)=∏t=1Tpθ(xt(v)∣zt(av),zt(v),w) \hspace{-.5cm} \small p_\theta\left(\myxa{} \mid \myzav{},\myza{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxa{t} \mid \myzav{t},\myza{t},\myw{}\right), \hspace{.3cm} \boxed{p_\theta\left(\myxv{} \mid \myzav{},\myzv{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxv{t} \mid \myzav{t},\myzv{t},\myw{}\right)}

pθ(z(a))=∏t=1Tpθ(zt(a)∣z1:t−1(a)),pθ(z(av))=∏t=1Tpθ(zt(av)∣z1:t−1(av)),pθ(z(v))=∏t=1Tpθ(zt(v)∣z1:t−1(v)) \hspace{-.5cm} \small p_\theta\left(\myza{}\right) = \prod_{t=1}^T p_\theta\left(\myza{t} \mid \myza{1:t-1} \right), \hspace{.3cm} p_\theta\left(\myzav{}\right) = \prod_{t=1}^T p_\theta\left(\myzav{t} \mid \myzav{1:t-1} \right), \hspace{.3cm} p_\theta\left(\myzv{}\right) = \prod_{t=1}^T p_\theta\left(\myzv{t} \mid \myzv{1:t-1} \right)

16

pθ(x(a)∣z(av),z(a),w)=∏t=1Tpθ(xt(a)∣zt(av),zt(a),w),pθ(x(v)∣z(av),z(v),w)=∏t=1Tpθ(xt(v)∣zt(av),zt(v),w) \hspace{-.5cm} \small p_\theta\left(\myxa{} \mid \myzav{},\myza{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxa{t} \mid \myzav{t},\myza{t},\myw{}\right), \hspace{.3cm} p_\theta\left(\myxv{} \mid \myzav{},\myzv{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxv{t} \mid \myzav{t},\myzv{t},\myw{}\right)

pθ(z(a))=∏t=1Tpθ(zt(a)∣z1:t−1(a)),pθ(z(av))=∏t=1Tpθ(zt(av)∣z1:t−1(av)),pθ(z(v))=∏t=1Tpθ(zt(v)∣z1:t−1(v)) \hspace{-.5cm} \small \boxed{p_\theta\left(\myza{}\right) = \prod_{t=1}^T p_\theta\left(\myza{t} \mid \myza{1:t-1} \right)}, \hspace{.3cm} \boxed{p_\theta\left(\myzav{}\right) = \prod_{t=1}^T p_\theta\left(\myzav{t} \mid \myzav{1:t-1} \right)}, \hspace{.3cm} \boxed{p_\theta\left(\myzv{}\right) = \prod_{t=1}^T p_\theta\left(\myzv{t} \mid \myzv{1:t-1} \right)}

16

pθ(x(a)∣z(av),z(a),w)=∏t=1Tpθ(xt(a)∣zt(av),zt(a),w),pθ(x(v)∣z(av),z(v),w)=∏t=1Tpθ(xt(v)∣zt(av),zt(v),w) \hspace{-.5cm} \small {p_\theta\left(\myxa{} \mid \myzav{},\myza{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxa{t} \mid \myzav{t},\myza{t},\myw{}\right)}, \hspace{.3cm} p_\theta\left(\myxv{} \mid \myzav{},\myzv{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxv{t} \mid \myzav{t},\myzv{t},\myw{}\right)

pθ(z(a))=∏t=1Tpθ(zt(a)∣z1:t−1(a)),pθ(z(av))=∏t=1Tpθ(zt(av)∣z1:t−1(av)),pθ(z(v))=∏t=1Tpθ(zt(v)∣z1:t−1(v)) \hspace{-.5cm} \small p_\theta\left(\myza{}\right) = \prod_{t=1}^T p_\theta\left(\myza{t} \mid \myza{1:t-1} \right), \hspace{.3cm} p_\theta\left(\myzav{}\right) = \prod_{t=1}^T p_\theta\left(\myzav{t} \mid \myzav{1:t-1} \right), \hspace{.3cm} p_\theta\left(\myzv{}\right) = \prod_{t=1}^T p_\theta\left(\myzv{t} \mid \myzv{1:t-1} \right)

These distributions are Gaussians parametrized by neural networks and p(w)=N(w;0,I) \small p(\myw{}) = \mathcal{N}(\myw{}; \mathbf{0}, \mathbf{I}).

16

Multimodal dynamical VAE (MDVAE)


  • Generative model
  • Inference model
  • Two-stage training
16

MDVAE inference model

  • As in the standard VAE, we need to define an inference model that approximates the posterior:

    qϕ(z(av),z(a),z(v),w∣x(a),x(v))≈pθ(z(av),z(a),z(v),w∣x(a),x(v)). q_\phi\left(\myzav{},\myza{},\myzv{}, \myw{} \mid \myxa{}, \myxv{}\right) \approx p_\theta\left(\myzav{},\myza{},\myzv{}, \myw{} \mid \myxa{}, \myxv{}\right).

  • The exact posterior is intractable, but using the chain rule, the Bayesian network of MDVAE and the D-separation principle (Geiger et al., 1990; Bishop, 2006), we can analyze the exact posterior dependencies, i.e. how the observed and latent variables depend on each other given the observations.

    See (Girin et al., 2021) for an extensive discussion of D-separation in the context of DVAEs.

L. Girin et al., Dynamical variational autoencoders: A comprehensive review, Foundations and Trends in Machine Learning, 2021.
D. Geiger et al., Identifying independence in Bayesian networks, Networks, 1990.
C. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

17

The inference model qϕ(z(av),z(a),z(v),w∣x(a),x(v))q_\phi\left(\myzav{},\myza{},\myzv{}, \myw{} \mid \myxa{}, \myxv{}\right) decomposes as the product of four terms:

qϕ(w∣x(a),x(v))×qϕ(z(av)∣x(a),x(v),w)×qϕ(z(a)∣x(a),z(av),w)×qϕ(z(v)∣x(v),z(av),w)\hspace{-.5cm} q_\phi\left(\myw{} \mid \myxa{}, \myxv{}\right) \times q_\phi\left(\myzav{}\mid \myxa{}, \myxv{}, \myw{} \right) \times q_\phi\left(\myza{} \mid \myxa{}, \myzav{}, \myw{}\right) \times q_\phi\left(\myzv{} \mid \myxv{}, \myzav{}, \myw{}\right)

18

The inference model qϕ(z(av),z(a),z(v),w∣x(a),x(v))q_\phi\left(\myzav{},\myza{},\myzv{}, \myw{} \mid \myxa{}, \myxv{}\right) decomposes as the product of four terms:

qϕ(w∣x(a),x(v))×qϕ(z(av)∣x(a),x(v),w)×qϕ(z(a)∣x(a),z(av),w)×qϕ(z(v)∣x(v),z(av),w)\hspace{-.5cm} \boxed{ q_\phi\left(\myw{} \mid \myxa{}, \myxv{}\right)} \times q_\phi\left(\myzav{}\mid \myxa{}, \myxv{}, \myw{} \right) \times q_\phi\left(\myza{} \mid \myxa{}, \myzav{}, \myw{}\right) \times q_\phi\left(\myzv{} \mid \myxv{}, \myzav{}, \myw{}\right)

18

The inference model qϕ(z(av),z(a),z(v),w∣x(a),x(v))q_\phi\left(\myzav{},\myza{},\myzv{}, \myw{} \mid \myxa{}, \myxv{}\right) decomposes as the product of four terms:

qϕ(w∣x(a),x(v))×qϕ(z(av)∣x(a),x(v),w)×qϕ(z(a)∣x(a),z(av),w)×qϕ(z(v)∣x(v),z(av),w)\hspace{-.5cm} q_\phi\left(\myw{} \mid \myxa{}, \myxv{}\right) \times \boxed{ q_\phi\left(\myzav{}\mid \myxa{}, \myxv{}, \myw{} \right) } \times q_\phi\left(\myza{} \mid \myxa{}, \myzav{}, \myw{}\right) \times q_\phi\left(\myzv{} \mid \myxv{}, \myzav{}, \myw{}\right)

18

The inference model qϕ(z(av),z(a),z(v),w∣x(a),x(v))q_\phi\left(\myzav{},\myza{},\myzv{}, \myw{} \mid \myxa{}, \myxv{}\right) decomposes as the product of four terms:

qϕ(w∣x(a),x(v))×qϕ(z(av)∣x(a),x(v),w)×qϕ(z(a)∣x(a),z(av),w)×qϕ(z(v)∣x(v),z(av),w)\hspace{-.5cm} q_\phi\left(\myw{} \mid \myxa{}, \myxv{}\right) \times q_\phi\left(\myzav{}\mid \myxa{}, \myxv{}, \myw{} \right) \times \boxed{ q_\phi\left(\myza{} \mid \myxa{}, \myzav{}, \myw{}\right) } \times \boxed{ q_\phi\left(\myzv{} \mid \myxv{}, \myzav{}, \myw{}\right) }

18

The inference model qϕ(z(av),z(a),z(v),w∣x(a),x(v))q_\phi\left(\myzav{},\myza{},\myzv{}, \myw{} \mid \myxa{}, \myxv{}\right) decomposes as the product of four terms:

qϕ(w∣x(a),x(v))×qϕ(z(av)∣x(a),x(v),w)×qϕ(z(a)∣x(a),z(av),w)×qϕ(z(v)∣x(v),z(av),w)\hspace{-.5cm} q_\phi\left(\myw{} \mid \myxa{}, \myxv{}\right) \times q_\phi\left(\myzav{}\mid \myxa{}, \myxv{}, \myw{} \right) \times q_\phi\left(\myza{} \mid \myxa{}, \myzav{}, \myw{}\right) \times q_\phi\left(\myzv{} \mid \myxv{}, \myzav{}, \myw{}\right)

To complete the inference model, we also need to look at the posterior temporal dependencies for the dynamical latent variables z(av)\myzav{}, z(a)\myza{} and z(v)\myzv{}.

18

Using again the chain rule, MDVAE Bayesian network and the D-separation principle, we have:

qϕ(z(av)∣x(a),x(v),w)=∏t=1Tqϕ(zt(av)∣z1:t−1(av), xt:T(a), xt:T(v), w)qϕ(z(a)∣x(a),z(av),w)=∏t=1Tqϕ(zt(a)∣z1:t−1(a), xt:T(a), zt(av), w)qϕ(z(v)∣x(v),z(av),w)=∏t=1Tqϕ(zt(v)∣z1:t−1(v), xt:T(v), zt(av), w) \begin{aligned} q_\phi\left(\myzav{}\mid \myxa{}, \myxv{}, \myw{} \right) & = \prod\limits_{t=1}^T q_\phi\left(\myzav{t} \mid \myzav{1:t-1},\, \myxa{t:T},\, \myxv{t:T},\, \myw{} \right) \\ q_\phi\left(\myza{} \mid \myxa{}, \myzav{}, \myw{}\right) & = \prod\limits_{t=1}^T q_\phi\left(\myza{t} \mid \myza{1:t-1},\, \myxa{t:T},\, \myzav{t},\, \myw{}\right) \\ q_\phi\left(\myzv{} \mid \myxv{}, \myzav{}, \myw{}\right) & = \prod\limits_{t=1}^T q_\phi\left(\myzv{t} \mid \myzv{1:t-1},\, \myxv{t:T},\, \myzav{t},\, \myw{}\right) \end{aligned}

These distributions and qϕ(w∣x(a),x(v))q_\phi\left(\myw{} \mid \myxa{}, \myxv{}\right) are Gaussians parametrized by neural networks.

Drawing the corresponding probabilistic graphical model at inference time would be difficult and not really informative.

19

Multimodal dynamical VAE (MDVAE)


  • Generative model
  • Inference model
  • Two-stage training
19

ELBO

As in the standard VAE, learning the MDVAE generative and inference model parameters consists in maximizing the ELBO

L(ϕ,θ)=Eqϕ(z(av), z(a), z(v), w∣x(a), x(v))[ln⁡pθ(x(a),x(v)∣z(av),z(a),z(v),w)]⏟reconstruction accuracy−DKL(qϕ(z(av),z(a),z(v),w∣x(a),x(v))∣∣ pθ(z(av),z(a),z(v),w))⏟regularization.\begin{aligned} \mathcal{L}(\phi, \theta) &= \underbrace{\mathbb{E}_{q_\phi\left(\myzav{},\, \myza{},\, \myzv{},\, \myw{} \mid \myxa{},\, \myxv{}\right)} \left[\ln p_\theta\left(\myxa{}, \myxv{} \mid \myzav{},\myza{},\myzv{}, \myw{}\right)\right]}_{\text{reconstruction accuracy}} \\ & \hspace{1cm} - \underbrace{D_{\text{KL}}\left(q_\phi\left(\myzav{},\myza{},\myzv{}, \myw{} \mid \myxa{}, \myxv{}\right) \Big\lvert\Big\rvert\, p_\theta\left(\myzav{},\myza{},\myzv{}, \myw{}\right) \right)}_{\text{regularization}}. \end{aligned}

Developing this expression is a bit more complicated than with the standard VAE, but there is no fundamental difficulty.

20

Two-stage training with the VQ-VAE

  • Standard VAEs tend to reconstruct blurred outputs, which is particularly true for image data.

  • The vector-quantized VAE (VQ-VAE) (van den Oord et al., 2017) learns a discrete latent representation to overcome this limitation.

    Before being fed to the decoder, the continuous latent vector is quantized using a discrete codebook that is jointly learned with the network architecture.

  • We exploit the VQ-VAE to train the MDVAE in two stages.

A. van den Oord et al., Neural discrete representation learning, NeurIPS 2017.

21

The first stage consists in learning a VQ-VAE independently for each modality and without temporal modeling.

Rather than learning the MDVAE on the raw audio and visual data, we will use the intermediate compressed representation of the VQ-VAEs before quantization.

22

In the second stage, we learn the MDVAE model "inside" the frozen VQ-VAE. This 2-stage training improves the reconstruction quality, but it also speeds up the training of the MDVAE model.

The disentanglement between static versus dynamical and modality-specific versus audiovisual latent speech factors occurs during this second training stage.

23

Experiments on audiovisual speech


  • Qualitative analysis of the learned representations
  • Audiovisual facial image denoising
  • Audiovisual speech emotion recognition
23

MEAD: Multi-view Emotional Audio-visual Dataset (K. Wang et al., 2020)

We use about 30 hours of audiovisual emotional speech from the MEAD dataset

  • 48 speakers (different for training and testing)
  • 8 different emotions
  • 3 levels of intensity
  • 7 views (we keep only the frontal view)

Preprocessing:

  • Face images are cropped, resized (64x64 resolution) and aligned.

Image credits: (K. Wang et al., 2020)

  • STFT parameters for computing power spectrograms are chosen such that the audio frame rate is equal to the visual frame rate (30 fps).

K. Wang et al., MEAD: A Large-scale Audio-visual Dataset for Emotional Talking-face Generation, ECCV, 2020

24

The first set of experiments consists in studying what characteristics of the audiovisual speech data are encoded in w\myw{}, z(av)\myzav{}, z(v)\myzv{}, and z(a)\myza{}.

We will present qualitative results obtained by reconstructing an audiovisual speech sequence using some of the latent variables from another sequence.

25

We transfer z(av)\myzav{} from the central sequence in red to the surrounding sequences.

Lip and jaw movements are transfered.

We transfer z(v)\myzv{} from the central sequence in red to the surrounding sequences.

Head and eyelid movements are transfered.

26

We transfer z(av)\myzav{} and z(v)\myzv{} from the central sequence in red to the surrounding sequences. The identity and global emotional state are preserved because w\myw{} is unaltered.

27

Interpolation of the static audiovisual latent variable w\myw{}

Same emotion, different identities.
Same identity, different emotions.

pθ(x(v)∣z(av),z(v),w)=∏t=1Tpθ(xt(v)∣zt(av),zt(v),w~t),w~t=αtw+(1−αt)w′,αt=(T−t)/(T−1). \small \hspace{-.5cm} p_\theta\left(\myxv{} \mid \myzav{},\myzv{},\myw{}\right) = \prod_{t=1}^T p_\theta\left(\myxv{t} \mid \myzav{t},\myzv{t},\boxed{\tilde{\myw{}}_t}\right), \hspace{.3cm} \tilde{\myw{}}_t = \alpha_t \myw{} + (1- \alpha_t) \myw{}', \hspace{.3cm} \alpha_t = (T-t)/(T-1).

28

  • The qualitative analysis confirmed that:

    • The static audiovisual latent variable w\myw{} encodes the speaker's identity and global emotional state.
    • The dynamical audiovisual latent variable z(av)\myzav{} encodes the speaker's lip and jaw movements.
    • The dynamical visual latent variable z(v)\myzv{} encodes the remaining facial movements such as the eyes and head movements.

  • These conclusions are confirmed quantitatively by measuring the impact of swapping latent variables on the action units (not presented today).

29

Experiments on audiovisual speech


  • Qualitative analysis of the learned representations
  • Audiovisual facial image denoising
  • Audiovisual speech emotion recognition
29
  • We artificially corrupt the visual modality by adding random Gaussian noise on localized regions of the 6 central frames of a 10 frame-long sequence. The audio modality is unaltered.

  • The task is to denoise the visual modality.

  • We compare MDVAE with two unimodal baselines trained on the visual modality only:

    • VQ-VAE (van den Oord et al., 2017), no temporal modeling.
    • DSAE (Li and Mandt, 2018), same temporal model as MDVAE and trained in two stages as MDVAE.

Denoising is done by simply encoding and decoding the corrupted (audio)visual speech sequence.

Y. Li and S. Mandt, "Disentangled sequential autoencoder", ICML 2018.

30

31

32


  • Results are obtained by averaging over 200 test sequences.
  • Metrics are computed on the corrupted region.

The performance gap between MDVAE and the unimodal baselines is larger for the corruption of the mouth region.

This is because MDVAE exploits the audio modality.

33

Experiments on audiovisual speech


  • Qualitative analysis of the learned representations
  • Audiovisual facial image denoising
  • Audiovisual speech emotion recognition
33

  • The qualitative analysis of the latent representations learned by MDVAE suggests that the static audiovisual latent variable w\myw{} encodes the speaker's emotion.

  • We propose to use the mean vector of the Gaussian inference model qϕ(w∣x(a),x(v))q_\phi\left(\myw{} \mid \myxa{}, \myxv{}\right) as the input of a multinomial logistic regression model trained for emotion classification on the MEAD dataset (8 classes).

  • The mean vector is simply obtained by a forward through the encoder network corresponding to qϕ(w∣x(a),x(v))q_\phi\left(\myw{} \mid \myxa{}, \myxv{}\right).

  • We compare the performance of MDVAE with its unimodal counterparts:

    • A-DSAE relies on the audio-only inference model qϕ(w∣x(a))q_\phi\left(\myw{} \mid \myxa{}\right);
    • V-DSAE relies on the visual-only inference model qϕ(w∣x(v))q_\phi\left(\myw{} \mid \myxv{}\right).
34

  • Using the exact same experimental protocol, MDVAE outperforms its two unimodal counterparts by about 50% of accuracy.
  • With less than 10% of the labeled data, MDVAE reaches 90% of its maximal performance.
35
  • We also evaluate the audiovisual emotion classification performance on RAVDESS (Livingstone and Russo, 2018).
  • We compare with a state-of-the-art model based on an audiovisual transformer architecture (Chumachenko et al., 2022).

F1 score (%) Precision (%) Recall (%)
Audiovisual transformer
(Chumachenko et al., 2022)		 88.65 89.37 87.96
MDVAE w/o finetuning
+ multinomial logisitic reg. 		82.86 81.98 83.76
MDVAE w/ finetuning
+ multinomial logisitic reg. 		89.62 89.55 89.71

Credits: (Chumachenko et al., 2022)

EfficientFace is pre-trained on AffectNet, the largest dataset of in-the-wild facial images labeled in emotions.

Finetuning MDVAE is unsupervised.


S.R Livingstone and F.A. Russo, "The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English", PloS one, 2018.
K. Chumachenko et al., "Self-attention fusion for audiovisual emotion recognition with incomplete data", arXiv preprint arXiv:2201.11095, 2022.

36

Conclusion

We proposed the MDVAE to learn structured representations of multimodal and dynamical data.

  • Why?

    Collecting labels for every scenario and tasks is intractable, we need alternatives to supervised learning.

  • How?

    Deep generative modeling is a powerful unsupervised learning paradigm that can be applied to many different types of data, in particular multimodal and sequential data.

    We can learn structured and interpretable representations by modeling probabilistic dependencies between observed and latent variables.

  • What?

    Various applications in audiovisual speech processing, using one single model.

37

MDVAE effectively combines the audio and visual information in static (w\myw{}) and dynamical (z(av)\myzav{}) latent variables:

  • Talking faces can be synthesized by transfering z(av)\myzav{} from one sequence to another, which preserves the speaker's identity, emotional state and visual-only facial movements.

  • The audio modality provides robstuness with respect to corruption of the visual modality on the mouth region.

  • The static audiovisual latent variable w\myw{} can be used for emotion recognition with few labeled data, and with much better accuracy compared with unimodal baselines.

    MDVAE is also competitive with a state-of-the-art method based on audiovisual transformers.

38

Extensions and applications of MDVAE include:

  • audiovisual-consistent data augmentation using pθ(z(v))p_\theta\left(\myzv{}\right) and/or p(w)p\left(\myw{}\right) for, e.g., automatic speech recognition;
  • deep speech prior for unsupervised audiovisual speech enhancement (Sadeghi et al., 2020);
  • audiovisual voice conversion, provided we improve the audio speech generative model, e.g., inspiring from RAVE (Caillon and Esling, 2021).

M. Sadeghi et al., "Audio-visual speech enhancement using conditional variational auto-encoders", IEEE/ACM Transactions on Audio, Speech and Language Processing, 2020
A. Caillon and P. Esling, "RAVE: A variational autoencoder for fast and high-quality neural audio synthesis", arXiv preprint arXiv:2111.05011, 2021

39

Thank you for your attention!

39

Joint work with


Samir Sadok1


Laurent Girin2


Xavier Alameda-Pineda3


Renaud Séguier1

1 CentraleSupélec, IETR (UMR CNRS 6164), France

2 Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France

3 Inria, Univ. Grenoble Alpes, CNRS, LJK, France

2
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, →, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow