In Generative model, we need to find joint distribution $p(x, y)$ and $$p(y | x) \propto p(x, y)$$

On the one hand, Discriminative models cannot generate objects. On the other hand, they are much more expressive because they do not make any internal assumptions about the structure of the object's feature space $X$.

Naive Bayes

$$ p(x, y) = p(y) \prod\limits_{i=1}^n p(x_i | y) $$

Autoregressive models (including Transformers)

$$ p(x_1, \dots, x_n) = p(x_1) p(x_2|x_1) \dots p(x_n | x_1, \dots, x_{n-1}) $$

$$ p(x) \approx q(x) = \dots, \quad KL(q \| p) \to \min $$

Diffusion based

$$ x \to x_1 = x + \varepsilon_1 \to \dots \to x_n \sim N(0, \Sigma) $$

Variational auto encoders (VAE) is in slides

Generative Adversarial Networks — GANs

$$ z \sim N(0, \Sigma) \to \boxed{\text{Model}} \to x$$

• Feature generation, for example, to effectively solve supervised learning problems
• Dimensionality reduction
• Low loss data compression
• Trainable object vectorization, embeddable in deeper neural network architectures
• Generation of synthetic objects similar to real ones

---

Rumelhart, Hinton, Williams. Learning Internal Representations by Error Propagation. 1986.

David Charte et al. A practical tutorial on autoencoders for nonlinear feature fusion: taxonomy, models, software and guidelines. 2018.

• it is not necessarily $B=A^T$ (although this is often done)
• arbitrary $A, B$ instead of orthogonal
• non-linear models
• arbitrary loss function $\mathscr{L}$ instead of quadratic
• SGD optimization instead of singular value decomposition (SVD)

• Applying $L_1$ or $L_2$ regularization to weight vectors
• Applying of $L_1$-regularization to representation vectors $z_i = Ax_i$
• Entropy regularization

---

D.Arpit et al. Why regularized auto-encoders learn sparse representation? 2015.

A generative model is constructed capable of generating new objects $x$ similar to the objects of the sample $X^\ell = \{x_1,\dots,x_\ell \}$

$q_\alpha(z|x)$ — probabilistic encoder with $\alpha$ parameter

$p_\beta(\hat x|z)$ — probabilistic decoder with $\beta$ parameter

$$ \begin{align*} \mathscr{L}_{VAE}(\alpha, \beta) = \sum\limits_{i=1}^\ell \log p(x_i) = \sum\limits_{i=1}^\ell \log \int q_{\alpha} (z | x_i) \frac{p_{\beta}(x_i|z) p(z)}{q_{\alpha} (z | x_i)} dz \geq \\ \geq \sum\limits_{i=1}^\ell \int q_\alpha(z|x_i) \log p_\beta(x_i|z)dz - KL(q_\alpha(z|x_i)\| p( z)) \to \max\limits_{\alpha, \beta} \end{align*} $$

Criteria:

• Discriminative model training $D$ (binary cross-entropy in essence): $$ -L_D = \sum\limits_{i=1}^m \ln D(x_i, {\color{orange}\beta}) + \ln(1 - D(G(z_i, \alpha), {\color{orange}\beta})) \to {\color{orange}\max\limits_{\beta}}$$
• Learning the generative model $G$ from random noise $\{z_i\}_{i=1}^m$: $$ L_G = \sum\limits_{i=1}^m \ln(1 - D(G(z_i, {\color{orange}\alpha}), \beta)) \to {\color{orange}\min\limits_{\alpha}}$$

It is not an easy question!

Usual SGD "as is" doesn't work here, therefore in first publications authors:

1. fix $\alpha$, train $\beta$
2. fix $\beta$, train $\alpha$

which is similar to EM algorithm.

And this also doesn't work in practice due to gradient vanishing. The first naive workaround is changing

$$ L_G = \sum\limits_{i=1}^m \ln(1 - D(G(z_i, {\color{orange}\alpha}), \beta)) \to L_G^\prime = -\sum\limits_{i=1}^m \ln(D(G(z_i, {\color{orange}\alpha}), \beta))$$

The EMD between probability distributions $P$ and $Q$ can be defined as an infimum over joint probabilities:

$$\text{EMD}(P,Q) = \inf\limits_{\gamma \in \Pi(P, Q)} \mathbb{E}_{(x, y) \sim \gamma}\left[d(x, y)\right]\,$$

where $\Pi(P, Q)$ is the set of all joint distributions whose marginals are $P$ and $Q$. By Kantorovich-Rubinstein duality, this can also be expressed as:

$$\text{EMD}(P,Q) = \sup\limits_{\| f \|_L \leq 1} \, \mathbb{E}_{x \sim P}[f(x)] - \mathbb{E}_{y \sim Q}[f(y)]\,$$

where the supremum is taken over all 1-Lipschitz continuous functions, i.e. $\| \nabla f(x)\| \leq 1 \quad \forall x$.

• Explored the distinction between Generative and Discriminative models in machine learning, highlighting their applications, strengths, and limitations
• Discussed Supervised Learning Task Formulation focusing on the probability distributions $p(y|x)$ for discriminative models and joint distribution $p(x,y)$ for generative models
• Delved into Generative Models, covering Explicit and Implicit Density Models, including Naive Bayes, Autoregressive Models, Variational Autoencoders (VAE), and Generative Adversarial Networks (GANs)
• Explored Autoencoders in depth, including their architecture, applications, and comparison with PCA. Also, discussed various types like Sparse and Variational Autoencoders
• Highlighted GANs and their development over time, including DCGAN, LSGAN, and Wasserstein GAN, showcasing their unique features and improvements in stability and image generation quality
• Reviewed advanced concepts such as the Wasserstein distance and its significance in GANs, alongside practical challenges and solutions in training GANs
• Concluded with demonstrations of GAN capabilities, including StyleGAN, and discussed metrics for evaluating generative models like Inception and FID scores