Functions of random variables

Assume $X$ is a random variable with distribution $p_x(x)$, then its function $Y=\phi(X)$ is also a random variable. We have $X=\phi^{-1}(Y)=\psi(Y)$ and $dy/dx=\phi'$ and $dx/dy=\psi'=1/\phi'$.

• Distribution $p_y(y)$ of $Y$ is related to distribution $p_x(x)$ of $X$:
  • If $Y=\phi(X)$ monotonically increases ($\phi'>0$ and $\psi'>0$), then
    
    $$P_y(y)=P(Y<y)=P(X<x)=\int_{-\infty}^x p_x(x)dx =\int_{-\infty}^{\psi(y)} p_x(\psi(y))dx$$
    
    
    
    
    $$p_y(y)=dP_y(y)/dy=p_x(\psi(y))\psi'(y)=p_x(x)/\phi'(x)$$
    
    

  • If $Y=\phi(X)$ monotonically decreases ($\phi'<0$ and $\psi'<0$), then
    
    $$P_y(y)=P(Y<y)=P(X>x)=\int_x^{\infty} p_x(x)dx =\int_{\psi(y)}^x p_x(\psi(y))dx$$
    
    
    
    
    $$p_y(y)=dP_y(y)/dy=-p_x(\psi(y))\psi'(y)=p_x(\psi(y))\vert\psi'(y)\vert=p_x(x)/\vert\phi'(x)\vert$$
    
    

  In general, we have
  
  $$p_y(y)=\sum_i p_x(x_i)/\vert\phi'(x_i)\vert$$
  
  
  where $x_i$ are solutions for equation $y=\phi(x)$.

• Entropy $H(Y)$ of $Y$ is related to entropy $H(X)$ of $X=\phi^{-1}(Y)=\psi(Y)$:

  
  

  $$H(Y) = -\int p_y(y) \; log \; p_y(y) dy =-\int \frac{p_x(x)}{\vert\phi'(x)\vert} log \; \frac{p_x(x)}{\vert\phi'(x)\vert} dy$$

  $$= -\int p_x(x) \; log \; \frac{p_x(x)}{\vert\phi'(x)\vert} dx =-\int p_x(x) \; log \; p_x(x) \; dx+\int p_x(x)\; log \; \vert\phi'(x)\vert dx$$

  $$= H(X)+E\; \{log\;\vert\phi'(X)\vert\}$$

  
  

If the inverse function $X=\psi^{-1}(Y)$ is not unique, than

$$H(Y)<H(X)+E\; \{log\;\vert\phi'(x)\vert\}$$

This result can be generalized to multi-variables. If

$$Y_i=\phi_i(X_1,\cdots,X_n),\;\;\;\;\;(i=1,\cdots,n)$$

then

$$H(Y_1,\cdots,Y_n) \le H(X_1,\cdots,X_n)+E\;\{ log\;J(X_1,\cdots,X_n)\}$$

where $J(X_1,\cdots,X_n)$ is the Jacobian of the above transformation:

$$J(X_1,\cdots,X_n)=\left\vert \begin{array}{ccc} \frac{\parti... ... &\frac{\partial \phi_n}{\partial X_n} \end{array} \right\vert$$

In particular, if the functions are linear

$$Y_i=\sum_{j=1}^n a_{ij} X_j,\;\;\;\;\;(i=1,\cdots,n)$$

then

$$H(Y_1,\cdots,Y_n) \le H(X_1,\cdots,X_n)+log\;det(A)$$

where $det(A)$ is the determinant of the transform matrix $A=[a_{ij}]_{n\times n}$. Again, the equation holds if the transform is unique.