两个多维高斯分布之间的KL散度推导

  在深度学习中,我们通常对模型进行抽样并计算与真实样本之间的损失,来估计模型分布与真实分布之间的差异。并且损失可以定义得很简单,比如二范数即可。但是对于已知参数的两个确定分布之间的差异,我们就要通过推导的方式来计算了。

  下面对已知均值与协方差矩阵的两个多维高斯分布之间的KL散度进行推导。当然,因为便于分布之间的逼近,Wasserstein distance可能是衡量两个分布之间差异的更好方式,但这个有点难,以后再记录。

  首先定义两个$n$维高斯分布如下:

$\begin{aligned} &p(x) = \frac{1}{(2\pi)^{0.5n}|\Sigma|^{0.5}}\exp\left(-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)\right)\\ &q(x) = \frac{1}{(2\pi)^{0.5n}|L|^{0.5}}\exp\left(-\frac{1}{2}(x-m)^T L^{-1}(x-m)\right)\\ \end{aligned}$

  需要计算的是:

$\begin{aligned} \text{KL}(p||q) = \text{E}_p\left(\log\frac{p(x)}{q(x)}\right) \end{aligned}$

  为了方便说明,下面分步进行推导。首先:

$\begin{aligned} \frac{p(x)}{q(x)} &= \frac {\frac{1}{(2\pi)^{0.5n}|\Sigma|^{0.5}}\exp\left(-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)\right)} {\frac{1}{(2\pi)^{0.5n}|L|^{0.5}}\exp\left(-\frac{1}{2}(x-m)^T L^{-1}(x-m)\right)}\\ &=\left(\frac{|L|}{|\Sigma|}\right)^{0.5}\exp\left(\frac{1}{2}(x-m)^T L^{-1}(x-m) -\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)\right) \end{aligned}$

  然后加上对数:

$\begin{aligned} \log\frac{p(x)}{q(x)} &= \frac{1}{2}\log\frac{|L|}{|\Sigma|}+ \frac{1}{2}(x-m)^T L^{-1}(x-m) – \frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu) \end{aligned}$

  再加上期望:

 $\begin{aligned} \text{E}_p\log\frac{p(x)}{q(x)} &=\frac{1}{2}\log\frac{|L|}{|\Sigma|}+ \text{E}_p\left[\frac{1}{2}(x-m)^T L^{-1}(x-m) – \frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)\right]\\ &=\frac{1}{2}\log\frac{|L|}{|\Sigma|}+ \text{E}_p\text{Tr}\left[\frac{1}{2}(x-m)^T L^{-1}(x-m) – \frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)\right]\\ \end{aligned}$

  第二步是因为结果为标量,可以转换为计算迹的形式。接着由迹的平移不变性得:

$\begin{align} &\frac{1}{2}\log\frac{|L|}{|\Sigma|}+ \text{E}_p\text{Tr} \left[ \frac{1}{2}L^{-1}(x-m)(x-m)^T – \frac{1}{2}\Sigma^{-1}(x-\mu)(x-\mu)^T \right]\\ = &\frac{1}{2}\log\frac{|L|}{|\Sigma|}+ \frac{1}{2}\text{E}_p\text{Tr} \left(L^{-1}(x-m)(x-m)^T\right) – \frac{1}{2}\text{E}_p\text{Tr} \left(\Sigma^{-1}(x-\mu)(x-\mu)^T\right) \\ = &\frac{1}{2}\log\frac{|L|}{|\Sigma|}+ \frac{1}{2}\text{E}_p\text{Tr} \left(L^{-1}(x-m)(x-m)^T\right) – \frac{n}{2} \end{align}$

  其中最后一项是因为,首先期望与迹可以调换位置,然后$(x-\mu)(x-\mu)^T$在分布$p$下的期望就是对应的协方差矩阵$\Sigma$,于是得到一个$n$维单位阵,再计算单位阵的迹为$n$。

  接下来,把中间项提出来推导,得:

$\begin{align} &\frac{1}{2}\text{E}_p\text{Tr} \left(L^{-1}(x-m)(x-m)^T\right)\\ =&\frac{1}{2}\text{Tr}\left(L^{-1}\text{E}_p \left(xx^T-xm^T-mx^T+mm^T \right) \right) \\ =&\frac{1}{2}\text{Tr}\left(L^{-1} \left(\Sigma +\mu\mu^T-2\mu m^T+mm^T \right) \right)  \end{align}$

  其中$\text{E}_p(xx^T) = \Sigma + \mu\mu^T$推导如下:

$\begin{aligned} \Sigma &= \text{E}_p\left[(x-\mu)(x-\mu)^T\right]\\ &= \text{E}_p\left(xx^T-x\mu^T-\mu x^T+\mu\mu^T\right)\\ &= \text{E}_p\left(xx^T\right)-2\text{E}_p\left(x\mu^T\right)+\mu\mu^T \\ &= \text{E}_p\left(xx^T\right)-\mu\mu^T \\ \end{aligned}$

  接着推导$(6)$式:

$\begin{aligned} &\frac{1}{2}\text{Tr}\left(L^{-1} \left(\Sigma +\mu\mu^T-2\mu m^T+mm^T \right) \right) \\ = &\frac{1}{2}\text{Tr}\left(L^{-1}\Sigma +L^{-1} (\mu-m)(\mu-m)^T \right) \\ = &\frac{1}{2}\text{Tr}\left(L^{-1}\Sigma\right)+ \frac{1}{2}(\mu-m)L^{-1}(\mu-m)^T  \\ \end{aligned}$

  最后代回$(3)$式,得到最终结果:

$\begin{aligned} \text{E}_p\log\frac{p(x)}{q(x)} =&\frac{1}{2}\left\{ \log\frac{|L|}{|\Sigma|}+ \text{Tr}\left(L^{-1}\Sigma\right)+ (\mu-m)L^{-1}(\mu-m)^T  – n \right\} \end{aligned}$

  参考于:两个多维高斯分布的Kullback-Leibler divergence(KL散度)