Discrete Mathematics

Derive sample mean update formula. Given $\mu_{n-1}=\frac{1}{n}\sum_{i=0}^{n-1}x_i$ , derive efficient calculation of $\mu_{n}$ .
Derive sample variance update formula. Given $\sigma_{n-1}^2=\frac{1}{n}\sum_{i=0}^{n-1}(x_i-\mu_{n-1})^2$ , derive efficient calculation of $\sigma_{n}^2$ .

Linear Algebra

Factor for $v-Av$
Proof that $(a-b)^T D (a-b)$ is a form of weighted mean squared error with weight on the diagonal matrix D
Define what frobenius norm is, what is the relation with Trace, and what is the derivative of frobenius norm wrt input matrix.

Vector Calculus and Probabilistic Modeling

Prove $(a-b)^2 = (b-a)^2$
Given $t=w_0+w_1x+\epsilon\quad \epsilon\sim\mathcal N(0, \sigma^2)$ .

Interpret $w_0$ correspond to the solution for maximum likelihood estimation given data $\mathcal D=\{(x_i, t_i)\}_i$

Given deterministic $y(x, w)$ and noise $\epsilon\sim\mathcal N(t\mid y(x, w), \sigma^2)$

t=y(x, w) + \epsilon

Find $w$ that maximize maximum likelihood given data $\mathcal D=\{(x_i, t_i)\}_i$

Offset Bias Weight Rule

\begin{align*} t &= \mathcal N(y, \sigma^2) \\ y &= w_0 + w_1x_0 + w_2x_1 + \dots + w_{n+1}x_n \end{align*}

Find the closed form of $w_0$ and interpret the function of it.

Probability to Closed Form

p(t\mid x, w) = \mathcal N(t\mid 3x+2, 1)

What is the assumed form of the data-generating model $t$ ?

Given Binary Outcomes with Bernoulli Likelihood

p(y=1\mid x, w) = \sigma(w^Tx)=\frac{1}{1+\exp(-w^Tx)}

What is the full expression for the likelihood $p(y\mid x, w)$ and naturally lead loss function.

Modeling Count Data with Poisson Likelihood

t\sim Poisson(\lambda(x))\quad \lambda(x)=\exp(w_0+w_1x)

Write the expression for $p(t\mid x,w)$ and best parameter $w$ using maximum likelihood estimation.

Deriving $\ell_2$ Least Square Solution

\frac{1}{2}\sum_i(t_i-w^T\phi(x_i))^2 + \frac{\lambda}{2}w^Tw

Solve for $w$ that minimize above expression

Multiple Gaussian Likelihood Problem

\begin{align*} p(\vec t\mid x, W, \sigma^2) &= \mathcal N(\vec t\mid W^T\phi(x), \sigma^2 I) \\ &= \frac{1}{\sqrt{(2\pi)^K\vert\sigma^2 I\vert}}\exp\left(-\frac{1}{2}(\vec t - W^T\phi(x))^T(\sigma I)^{-1}(\vec t - W^T\phi(x))\right) \end{align*}

Find the likelihood of above gaussian given $\mathcal D=\{(x_i, t_i)\}_i$ then find the best $w$ that maximize. What optimization we could use to solve for batch dataset (e.g. using the whole dataset instead of using one by one).

Multiple Regression Problem

Given $T=[t_i]_i\in R^{N\times K}, X=[\phi(x_i)]_i\in R^{N\times D}, W=[w^T_i]_i \in R^{D\times K}$ , find $w$ that minimize

\mathcal L(w)= \frac{1}{2}\sum_i^N \sum_j^K (w^T\phi(x_i)_j - t_{ij})^2

Express the above into matrix form utilizing forbenius norm and then solve for $w$ that minimize above expression.

Decision Theory in Classification

Given observation state age $x_0$ (old or young) and covid test result $x_1$ . Given prior assumption of covid $p(y)=0.1$ and $p(x_1=1\mid y=1)=0.875$ and $p(x_1=0\mid y=0)=0.975$ (no matter the age). Given $\ell(x, a)$ as below.

State	Do nothing	Give drug
No COVID-19, young	0	8
COVID-19, young	60	8
No COVID-19, old	0	8
COVID-19, old	10	8

For each state, find the optimal action $a^*$ that minimize the expected loss.

Cost Sensitive Learning

Given $\ell(y, a)$ as below. Assume we have a model $P(y\mid x)$

Predicted label	a = 0	a = 1
True label
y = 0	$\ell_{00}$	$\ell_{01}$
y = 1	$\ell_{10}$	$\ell_{11}$

Find the optimal decision rule $a^*$ that minimize the expected loss using our model $P$ .

Reject Decision Theory

Given $\ell(y, a)$ below, find the optimal decision rule $a^*$ that minimize the expected loss given model $P(Y\mid X)$ .

\ell l(y, a) = \begin{cases} 0&\text{if }a=\hat y\\ \lambda_r&\text{if }a=\text{reject}\\ \lambda_e&\text{otherwise} \end{cases}

MLE for Logistic Regression

Find closed form solution $w^*$ that maximize MLE for model below on dataset $\mathcal D=\{(x_i, y_i)\}_i$ .

p(y=1\mid x) = \frac{1}{1+\exp(-w^Tx+b)}

MLE for Multinomial Softmax

Find closed form solution $w^*$ that maximize MLE for model below on dataset $\mathcal D=\{(x_i, y_i)\}_i$ .

p(y=k\mid x) = \mathcal S(XW_n)