Appendix A

Appendix A

Proof that the solution to is $f(X_i)=E[Y_i \mid X_i]$.

Appendix B

Proof that the solution to $

is $\square$ Let

We solve: $$ \begin{aligned} &\min_{\beta_1, \beta_2} L
&\leftrightarrow

\begin{cases} \frac{dL}{d \beta_1} =0 \qquad \text{Call this }(a) \

\frac{dL}{d \beta_2} =0 \qquad \text{Call this }(b) \end{cases} \

&\leftrightarrow \begin{cases} \sum_{i=1}^n(\frac{d(y_i-\beta_1-\beta_2x_i)^2}{d \beta_1}) =0 \qquad \qquad \text{derivative of a sum}
(b)\end{cases} \

&\leftrightarrow \begin{cases}\sum_{i=1}^n-2(y_i-\beta_1-\beta_2x_i) =0 \

(b)\end{cases}\&\leftrightarrow \begin{cases}\beta_1 = \bar{y} - \beta_2\bar{x} \

(b)\end{cases}\\ \

&\leftrightarrow \begin{cases}(a) \

\sum_{i=1}^n(\frac{d(y_i-\beta_1-\beta_2x_i)^2}{d \beta_2}) =0 \qquad \qquad \text{derivative of a sum}\end{cases} \

&\leftrightarrow \begin{cases}(a)\\sum_{i=1}^n-2x_i(y_i-\beta_1-\beta_2x_i) =0\end{cases} \

&\leftrightarrow \begin{cases}(a)\\sum_{i=1}^n-2x_i(y_i-\bar{y}+\beta_2\bar{x}-\beta_2x_i) =0 \qquad \qquad \text{substitute }\beta_1

\end{cases} \

&\leftrightarrow \begin{cases}(a)\\sum_{i=1}^n x_i(y_i-\bar{y}+\beta_2\bar{x}-\beta_2x_i) =0\qquad \qquad \text{divide by -2}\end{cases} \

&\leftrightarrow \begin{cases}(a) \

\sum_{i=1}^n x_i(y_i-\bar{y}) +\sum_{i=1}^n \beta_2 x_i(\bar{x} - x_i) =0 \qquad \qquad \text{rearrange}\end{cases} \

&\leftrightarrow \begin{cases}(a) \

\sum_{i=1}^n x_i(y_i-\bar{y}) =\sum_{i=1}^n \beta_2 x_i(x_i - \bar{x}) \qquad \qquad \text{rearrange}\end{cases} \

&\leftrightarrow \begin{cases}(a) \

\sum_{i=1}^n (x_i-\bar{x})(y_i-\bar{y}) =\sum_{i=1}^n \beta_2 (x_i - \bar{x})^2 \qquad \qquad \text{by fact A} \end{cases} \

&\leftrightarrow \begin{cases}(a) \

       \beta_2  = \frac{\sum_{i=1}^n(y_i-\bar{y})(x_i- \bar{x})}{\sum_{i=1}^n  (x_i- \bar{x})^2} \qquad \qquad \text{rearrange}\end{cases}


      \\ \\

      &\leftrightarrow \begin{cases}\beta_1 = \bar{y} - \beta_2\bar{x} \\

       \beta_2  = \frac{\sum_{i=1}^n(y_i-\bar{y})(x_i- \bar{x})}{\sum_{i=1}^n  (x_i- \bar{x})^2} \end{cases} \end{aligned} $$

Thus we write: $$ \begin{aligned} \hat{\beta_1} &= \bar{y} - \hat{\beta_2}\bar{x} \

\hat{\beta_2} &= \frac{\sum_{i=1}^n(y_i-\bar{y})(x_i- \bar{x})}{\sum_{i=1}^n (x_i- \bar{x})^2} \ \ \blacksquare \end{aligned} $$

Fact A is proven here: $$ \begin{aligned} \sum_{i=1}^n (x_i-\bar{x})(y_i-\bar{y}) &= \sum_{i=1}^n(x_iy_i - x_i\bar{y} -\bar{x}y_i +\bar{x}\bar{y})
&= \sum_{i=1}^n(x_iy_i) -\bar{y}\sum_{i=1}^n(x_i) - \bar{x}\sum_{i=1}^n(y_i) + n\bar{x}\bar{y}
&= \sum_{i=1}^n(x_iy_i) -n\bar{y}\bar{x} - n\bar{y}\bar{x} + n\bar{x}\bar{y}
&= \sum_{i=1}^n(x_iy_i) -n\bar{y}\bar{x} \

&= \sum_{i=1}^n(x_iy_i) -\sum_{i=1}^ny_i\bar{x} & &= \sum_{i=1}^n(x_iy_i) -\sum_{i=1}^nx_i\bar{y} \

&= \sum_{i=1}^n y_i(x_i-\bar{x}) \qquad\text{factorise y_i} & &= \sum_{i=1}^n x_i(y_i-\bar{y}) \qquad\text{factorise x_i}
\end{aligned} $$

A special case of fact A is:

Appendix C

Proof that Where

and where

$\square$ Then

We can also write: Since $(X\beta)’Y$ is a scalar, it is equal to its transpose $Y’X\beta$. Thus:

We then solve: $$ \begin{aligned} \min_{\beta} U’U &\leftrightarrow

\frac{dU’U}{d\beta}=0
& \leftrightarrow

\frac{dY’Y }{d\beta} -2\frac{d\ Y’X\beta}{d\beta}+ \frac{d\beta’X’X\beta}{d\beta} =0
& \leftrightarrow 0 -2(Y’X)’ + (X’X+(X’X)’)\beta =0
& \leftrightarrow -2X’Y + 2X’X\beta =0 \

& \leftrightarrow X’X\beta = X’Y
\end{aligned} \begin{aligned} \min_{\beta} U’U & \leftrightarrow \beta = (X’X)^{-1} X’Y \ \ \blacksquare
\end{aligned}

$$