Today we finish 5.2 and start 5.3. Read Sections 5.3 and 5.4 for next class. Work through recommended homework questions.
Tutorials: Quiz 9 covers 4.6, 5.1 and the first part of 5.2 (orthogonal complements).
Office hour: Wed 11:30-noon, MC103B.
Help Centers: Monday-Friday 2:30-6:30 in MC106.
Final exam: Covers whole course, with an emphasis on the material after the midterm. Our course will end with Section 5.4. I will try to post some practice problems soon.
Question: If $W = \R^n$, then $W^\perp = \query{\{ \vec 0 \}}$
T/F: An orthogonal basis $\{ \vv_1, \ldots, \vv_k \}$ must have $$ \vv_i \cdot \vv_j = \begin{cases} 0 & \text{if } i \neq j \\ 1 & \text{if } i = j \end{cases} $$
Definition: Let $W$ be a subspace of $\R^n$. A vector $\vv$ is orthogonal to $W$ if $\vv$ is orthogonal to every vector in $W$. The orthogonal complement of $W$ is the set of all vectors orthogonal to $W$ and is denoted $W^\perp$. So $$ \kern-4ex W^\perp = \{ \vv \in \R^n : \vv \cdot \vw = 0 \text{ for all } \vw \text{ in } W \} $$
An example to keep in mind is where $W$ is a plane through the origin in $\R^3$ and $W^\perp$ is $\span(\vn)$, where $\vn$ is the normal vector to $W$.
Theorem 5.9: Let $W$ be a subspace of $\R^n$. Then:
a. $W^\perp$ is a subspace of $\R^n$.
b. $(W^\perp)^\perp = W$
c. $W \cap W^\perp = \{ \vec 0 \}$
d. If $W = \span(\vw_1, \ldots, \vw_k)$, then $\vv$ is in $W^\perp$
if and only if $\vv \cdot \vw_i = 0$ for all $i$.
We proved all of these except part (b), which will come today.
Theorem 5.10: Let $A$ be an $m \times n$ matrix. Then $$ \kern-4ex (\row(A))^\perp = \null(A) \qtext{and} (\col(A))^\perp = \null(A^T) $$
The first two are in $\R^n$ and the last two are in $\R^m$. These are the four fundamental subspaces of $A$.
If we write $W = \span(\vu)$, then $\vw = \proj_{\vu}(\vv)$ is in $W$,
$\vw^\perp = \Perp_{\vu}(\vv)$ is in $W^\perp$, and $\vv = \vw + \vw^\perp$.
We can do this more generally:
Definition: Let $W$ be a subspace of $\R^n$ and let $\{ \vu_1, \ldots, \vu_k \}$ be an orthogonal basis for $W$. For $\vv$ in $\R^n$, the orthogonal projection of $\vv$ onto $W$ is the vector $$ \kern-8ex \proj_W(\vv) = \proj_{\vu_1}(\vv) + \cdots + \proj_{\vu_k}(\vv) $$ The component of $\vv$ orthogonal to $W$ is the vector $$ \kern-8ex \Perp_W(\vv) = \vv - \proj_W(\vv) $$
We will show soon that $\Perp_W(\vv)$ is in $W^\perp$.
Note that multiplying $\vu$ by a scalar in the earlier example doesn't change $W$, $\vw$ or $\vw^\perp$. We'll see later that the general definition also doesn't depend on the choice of orthogonal basis.
Theorem: $\Perp_W(\vv)$ is in $W^\perp$.
Explain on board.
Now we will see that $\proj$ and $\Perp$ don't depend on the choice of orthogonal basis. Here and in the rest of the section, we assume that every subspace has at least one orthogonal basis.
Theorem 5.11: Let $W$ be a subspace of $\R^n$ and let $\vv$ be a vector in $\R^n$. Then there are unique vectors $\vw$ in $W$ and $\vw^\perp$ in $W^\perp$ such that $\vv = \vw + \vw^\perp$.
Proof: We saw above that such a decomposition exists, by taking $\vw = \proj_W(\vv)$ and $\vw^\perp = \Perp_W(\vv)$, using an orthogonal basis for $W$.
We now show that this decomposition is unique. So suppose $\vv = \vw_1 + \vw_1^\perp$ is another such decomposition. Then $\vw + \vw^\perp = \vw_1 + \vw_1^\perp$, so $$\vw - \vw_1 = \vw_1^\perp - \vw^\perp$$ The left hand side is in $W$ and the right hand side is in $W^\perp$ (why?), so both sides must be zero (why?). So $\vw = \vw_1$ and $\vw^\perp = \vw_1^\perp$.$\quad\Box$
Note that $\perp$ is an operation on subspaces, but is not an operation on vectors.
Now we can prove part (b) of Theorem 5.9.
Corollary 5.12: If $W$ is a subspace of $\R^n$, then $(W^\perp)^\perp = W$.
Proof: If $\vw$ is in $W$ and $\vx$ is in $W^\perp$, then $\vw \cdot \vx = 0$. This means that $\vw$ is in $(W^\perp)^\perp$. So $W \subseteq (W^\perp)^\perp$.
We need to show that every vector in $(W^\perp)^\perp$ is in $W$. So let $\vv$ be a vector in $(W^\perp)^\perp$. By the previous result, we can write $\vv$ as $\vw + \vw^\perp$, where $\vw$ is in $W$ and $\vw^\perp$ is in $W^\perp$. Then $$ \kern-4ex \begin{aligned} 0\ &= \vv \cdot \vw^\perp = (\vw + \vw^\perp)\cdot\vw^\perp \\ &= \vw \cdot \vw^\perp + \vw^\perp \cdot \vw^\perp = 0 + \vw^\perp \cdot \vw^\perp = \vw^\perp \cdot \vw^\perp \end{aligned} $$ So $\vw^\perp = \vec 0$ and $\vv = \vw$ is in $W$.$\quad\Box$
This next result is related to the Rank Theorem:
Theorem 5.13: If $W$ is a subspace of $\R^n$, then $$ \dim W + \dim W^\perp = n $$
Proof: Let $\{ \vu_1, \ldots, \vu_k \}$ be an orthogonal basis of $W$ and let $\{ \vv_1, \ldots, \vv_\ell \}$ be an orthogonal basis of $W^\perp$. Then $\{ \vu_1, \ldots, \vu_k, \vv_1, \ldots, \vv_\ell \}$ is an orthogonal basis for $\R^n$. (Explain.) The result follows.$\quad\Box$
Example: For $W$ a plane in $\R^3$, $2 + 1 = 3$.
The Rank Theorem follows if we take $W = \row(A)$, since then $W^\perp = \null(A)$:
Corollary 5.14 (The Rank Theorem, again): If $A$ is an $m \times n$ matrix, then $$ \rank(A) + \nullity(A) = n $$
Note: The logic here can be reversed. We can use the rank theorem to prove Theorem 5.13, and Theorem 5.13 can be used to prove Corollary 5.12.
Example: Let $W = \span(\vx_1, \vx_2)$ where $\vx_1 = \colll 1 1 0$ and $\vx_2 = \colll {-2} 0 1$. Find an orthogonal basis for $W$.
Solution: Ideas? Do on board.
Question: What if we had a third basis vector $\vx_3$?
Theorem 5.15 (The Gram-Schmidt Process): Let $\{ \vx_1, \ldots, \vx_k \}$ be a basis for a subspace $W$ of $\R^n$. Write $W_1 = \span(\vx_1)$, $W_2 = \span(\vx_1, \vx_2)$, $\ldots$, $W_k = \span(\vx_1, \ldots, \vx_k)$. Define: $$ \kern-9ex \begin{aligned} \vv_1\ &= \vx_1 \\ \vv_2\ &= \Perp_{W_1}(\vx_2) = \vx_2 - \frac{\vv_1 \cdot \vx_2}{\vv_1 \cdot \vv_1} \vv_1 \\ \vv_3\ &= \Perp_{W_2}(\vx_3) = \vx_3 - \frac{\vv_1 \cdot \vx_3}{\vv_1 \cdot \vv_1} \vv_1 - \frac{\vv_2 \cdot \vx_3}{\vv_2 \cdot \vv_2} \vv_2 \\ &\vdots \\ \vv_k\ &= \Perp_{W_{k-1}}(\vx_k) = \vx_k - \frac{\vv_1 \cdot \vx_k}{\vv_1 \cdot \vv_1} \vv_1 - \cdots - \frac{\vv_{k-1} \cdot \vx_k}{\vv_{k-1} \cdot \vv_{k-1}} \vv_{k-1} \end{aligned} $$ Then for each $i$, $\{ \vv_1, \ldots, \vv_i \}$ is an orthogonal basis for $W_i$. In particular, $\{ \vv_1, \ldots, \vv_k \}$ is an orthogonal basis for $W = W_k$.
Explain verbally.
Example 5.13: Apply Gram-Schmidt to construct an orthogonal basis for the subspace $W = \span(\vx_1, \vx_2, \vx_3)$ of $\R^4$ where $$ \vx_1 = \collll 1 {-1} {-1} 1, \quad \vx_2 = \collll 2 {1} {0} 1, \quad \vx_3 = \collll 2 {2} {1} 2 $$ On board, scaling intermediate results. We get $$ \vv_1 = \collll 1 {-1} {-1} 1, \quad \vv'_2 = \collll 3 3 1 1, \quad \vv'_3 = \collll {-1} 0 1 2 $$ If we want an orthonormal basis, we scale these: $$ \kern-9ex \vq_1 = {\textstyle\frac{1}{\|\vv_1\|}} \vv_1 = \frac{1}{2} \! \collll 1 {\!-1} {\!-1} 1, \ \vq_2 = {\textstyle\frac{1}{\|\vv'_2\|}} \vv'_2 = \frac{1}{\sqrt{20}}\! \collll 3 3 1 1, $$ $$ \kern-9ex \vq_3 = {\textstyle\frac{1}{\|\vv'_3\|}} \vv'_3 = \frac{1}{\sqrt{6}} \! \collll {\!-1} 0 1 2 $$
Notes: To compute $\Perp_{W_i}$ you have to use the orthogonal basis of $\vv_j$'s that you have constructed already, not the original basis of $\vx_j$'s.
The basis you get depends on the order of the vectors you start with. You should always do a question using the vectors in the order given, since that order will be chosen to minimize the arithmetic.
If you are asked to find an orthonormal basis, normalize each $\vv_j$ at the end. (It is correct to normalize earlier, but can be messier.)
You don't need to check that the starting vectors are linearly independent. If they are dependent, then one or more of the $\vv_j$'s will be $\vec 0$, and you can just ignore it.