07_linear-algebra.qmd

# Linear Algebra {#sec-linearalgebra}

Topics:

-   Working with Vectors
-   Linear Independence
-   Basics of Matrix Algebra
-   Square Matrices
-   Linear Equations
-   Systems of Linear Equations
-   Systems of Equations as Matrices
-   Solving Augmented Matrices and Systems of Equations
-   Rank
-   The Inverse of a Matrix
-   Inverse of Larger Matrices

## Working with Vectors {#sec-vector-def}

**Vector**: A vector in $n$-space is an ordered list of $n$ numbers. These numbers can be represented as either a row vector or a column vector: $$ {\bf v} \begin{pmatrix} v_1 & v_2 & \dots & v_n\end{pmatrix} , {\bf v} = \begin{pmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{pmatrix}$$

We can also think of a vector as defining a point in $n$-dimensional space, usually ${\bf R}^n$; each element of the vector defines the coordinate of the point in a particular direction.

**Vector Addition and Subtraction**: If two vectors, ${\bf u}$ and ${\bf v}$, have the same length (i.e. have the same number of elements), they can be added (subtracted) together: $$ {\bf u} + {\bf v} = \begin{pmatrix} u_1 + v_1 & u_2 + v_2 & \cdots & u_k + v_n \end{pmatrix}$$ $$ {\bf u} - {\bf v} = \begin{pmatrix} u_1 - v_1 & u_2 - v_2 & \cdots & u_k - v_n \end{pmatrix}$$

**Scalar Multiplication**: The product of a scalar $c$ (i.e. a constant) and vector ${\bf v}$ is:\
$$ c{\bf v} =  \begin{pmatrix} cv_1 & cv_2 & \dots & cv_n \end{pmatrix} $$

**Vector Inner Product**: The inner product (also called the dot product or scalar product) of two vectors ${\bf u}$ and ${\bf v}$ is again defined if and only if they have the same number of elements $$ {\bf u} \cdot {\bf v} = u_1v_1 + u_2v_2 + \cdots + u_nv_n = \sum_{i = 1}^n u_iv_i$$ If ${\bf u} \cdot {\bf v} = 0$, the two vectors are orthogonal (or perpendicular).

**Vector Norm**: The norm of a vector is a measure of its length. There are many different ways to calculate the norm, but the most common is the Euclidean norm (which corresponds to our usual conception of distance in three-dimensional space): $$ ||{\bf v}|| = \sqrt{{\bf v}\cdot{\bf v}} = \sqrt{ v_1v_1 + v_2v_2 + \cdots + v_nv_n}$$

::: {#exm-vectors}
## Vector Algebra

Let $a = \begin{pmatrix} 2&1&2\end{pmatrix}$, $b = \begin{pmatrix} 3&4&5 \end{pmatrix}$. Calculate the following:

1.  $a - b$

2.  $a \cdot b$
:::

::: {#exr-vectors1}
## Vector Algebra

Let $u = \begin{pmatrix} 7&1&-5&3\end{pmatrix}$, $v = \begin{pmatrix} 9&-3&2&8 \end{pmatrix}$, $w = \begin{pmatrix} 1&13& -7&2 &15 \end{pmatrix}$, and $c = 2$. Calculate the following:

1.  $u-v$

2.  $cw$

3.  $u \cdot v$

4.  $w \cdot v$
:::

## Linear Independence {#sec-linearindependence}

**Linear combinations**: The vector ${\bf u}$ is a linear combination of the vectors ${\bf v}_1, {\bf v}_2,  \cdots , {\bf v}_k$ if $${\bf u} = c_1{\bf v}_1 + c_2{\bf v}_2 +  \cdots + c_k{\bf v}_k$$

For example, $\begin{pmatrix}9 & 13 & 17 \end{pmatrix}$ is a linear combination of the following three vectors: $\begin{pmatrix}1 & 2 & 3 \end{pmatrix}$, $\begin{pmatrix} 2 & 3& 4\end{pmatrix}$, and $\begin{pmatrix} 3 & 4 & 5 \end{pmatrix}$. This is because $\begin{pmatrix}9 & 13 & 17 \end{pmatrix}$ = $(2)\begin{pmatrix}1 & 2 & 3 \end{pmatrix}$ $+ (-1)\begin{pmatrix} 2 & 3& 4\end{pmatrix}$ + $3\begin{pmatrix} 3 & 4 & 5 \end{pmatrix}$

**Linear independence**: A set of vectors ${\bf v}_1, {\bf v}_2,  \cdots , {\bf v}_k$ is linearly independent if the only solution to the equation $$c_1{\bf v}_1 + c_2{\bf v}_2 +  \cdots + c_k{\bf v}_k = 0$$ is $c_1 = c_2 = \cdots = c_k = 0$. If another solution exists, the set of vectors is linearly dependent.

A set $S$ of vectors is linearly dependent if and only if at least one of the vectors in $S$ can be written as a linear combination of the other vectors in $S$.

Linear independence is only defined for sets of vectors with the same number of elements; any linearly independent set of vectors in $n$-space contains at most $n$ vectors.

Since $\begin{pmatrix}9 & 13 & 17 \end{pmatrix}$ is a linear combination of $\begin{pmatrix}1 & 2 & 3 \end{pmatrix}$, $\begin{pmatrix} 2 & 3& 4\end{pmatrix}$, and $\begin{pmatrix} 3 & 4 & 5 \end{pmatrix}$, these 4 vectors constitute a linearly dependent set.

::: {#exm-linearindep}
## Linear Independence

Are the following sets of vectors linearly independent?

1.  $\begin{pmatrix}2 & 3 & 1 \end{pmatrix}$ and $\begin{pmatrix}4 & 6 & 1 \end{pmatrix}$
2.  $\begin{pmatrix}1 & 0 & 0 \end{pmatrix}$, $\begin{pmatrix}0 & 5 & 0 \end{pmatrix}$, and $\begin{pmatrix}10 & 10 & 0 \end{pmatrix}$
:::

::: {#exr-linearindep1}
## Linear Independence

Are the following sets of vectors linearly independent?

1.  $${\bf v}_1 = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix} , {\bf v}_2 = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} , {\bf v}_3 = \begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix} $$

2.  $${\bf v}_1 = \begin{pmatrix} 2 \\ 2 \\ 1 \end{pmatrix} , {\bf v}_2 = \begin{pmatrix} -4 \\ 6 \\ 5 \end{pmatrix} , {\bf v}_3 = \begin{pmatrix} -2 \\ 8 \\ 6 \end{pmatrix} $$
:::

## Basics of Matrix Algebra {#sec-matrixbasics}

**Matrix**: A matrix is an array of real numbers arranged in $m$ rows by $n$ columns. The dimensionality of the matrix is defined as the number of rows by the number of columns, $m \times n$.

$${\bf A}=\begin{pmatrix}
            a_{11} & a_{12} & \cdots & a_{1n} \\
            a_{21} & a_{22} & \cdots & a_{2n} \\
            \vdots & \vdots & \ddots & \vdots \\
            a_{m1} & a_{m2} & \cdots & a_{mn}
        \end{pmatrix}$$

Note that you can think of vectors as special cases of matrices; a column vector of length $k$ is a $k \times 1$ matrix, while a row vector of the same length is a $1 \times k$ matrix.

It's also useful to think of matrices as being made up of a collection of row or column vectors. For example, $$\bf A = \begin{pmatrix} {\bf a}_1 & {\bf a}_2 &  \cdots & {\bf a}_m \end{pmatrix}$$

**Matrix Addition**: Let $\bf A$ and $\bf B$ be two $m\times n$ matrices. $${\bf A+B}=\begin{pmatrix}
            a_{11}+b_{11} & a_{12}+b_{12} & \cdots & a_{1n}+b_{1n} \\
            a_{21}+b_{21} & a_{22}+b_{22} & \cdots & a_{2n}+b_{2n} \\
            \vdots & \vdots  & \ddots & \vdots \\
            a_{m1}+b_{m1} & a_{m2}+b_{m2} & \cdots & a_{mn}+b_{mn}
        \end{pmatrix}$$

Note that matrices ${\bf A}$ and ${\bf B}$ must have the same dimensionality, in which case they are **conformable for addition**.

::: {#exm-matrixaddition}
$${\bf A}=\begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix}, \qquad
            {\bf B}=\begin{pmatrix} 1 & 2 & 1 \\ 2 & 1 & 2 \end{pmatrix}$$ ${\bf A+B}=$
:::

**Scalar Multiplication**: Given the scalar $s$, the scalar multiplication of $s {\bf A}$ is $$ s {\bf A}=  s \begin{pmatrix}
            a_{11} & a_{12} & \cdots & a_{1n} \\
            a_{21} & a_{22} & \cdots & a_{2n} \\
            \vdots & \vdots & \ddots & \vdots \\
            a_{m1} & a_{m2} & \cdots & a_{mn}
        \end{pmatrix}
        = \begin{pmatrix}
            s a_{11} & s a_{12} & \cdots & s a_{1n} \\
            s a_{21} & s a_{22} & \cdots & s a_{2n} \\
            \vdots & \vdots & \ddots & \vdots \\
            s a_{m1} & s a_{m2} & \cdots & s a_{mn}
        \end{pmatrix}$$

::: {#exm-scalarmulti}
$s=2, \qquad {\bf A}=\begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix}$

$s {\bf A} =$
:::

**Matrix Multiplication**: If ${\bf A}$ is an $m\times k$ matrix and $\bf B$ is a $k\times n$ matrix, then their product $\bf C = A B$ is the $m\times n$ matrix where $$c_{ij}=a_{i1}b_{1j}+a_{i2}b_{2j}+\cdots+a_{ik}b_{kj}$$

::: {#exm-matrixmulti}
1.  $\begin{pmatrix} a&b\\c&d\\e&f \end{pmatrix} \begin{pmatrix} A&B\\C&D \end{pmatrix}
            =$

2.  $\begin{pmatrix} 1&2&-1\\3&1&4 \end{pmatrix} \begin{pmatrix} -2&5\\4&-3\\2&1\end{pmatrix} =$
:::

Note that the number of columns of the first matrix must equal the number of rows of the second matrix, in which case they are **conformable for multiplication**. The sizes of the matrices (including the resulting product) must be $$(m\times k)(k\times n)=(m\times n)$$

Also note that if **AB** exists, **BA** exists only if $\dim({\bf A}) = m \times n$ and $\dim({\bf B}) = n \times m$.

This does not mean that **AB** = **BA**. **AB** = **BA** is true only in special circumstances, like when ${\bf A}$ or ${\bf B}$ is an identity matrix or ${\bf A} = {\bf B}^{-1}$.

**Laws of Matrix Algebra**:

```{=tex}
\begin{enumerate}
        \item \parbox[t]{1.5in}{Associative:} $\bf (A+B)+C = A+(B+C)$\\
            \parbox[t]{1.5in}{\quad}  $\bf (AB)C = A(BC)$
        \item \parbox[t]{1.5in}{Commutative:} $\bf A+B=B+A$
        \item \parbox[t]{1.5in}{Distributive:} $\bf A(B+C)=AB+AC$\\
            \parbox[t]{1.5in}{\quad}   $\bf (A+B)C=AC+BC$
\end{enumerate}
```
Commutative law for multiplication does not hold -- the order of multiplication matters: $$\bf AB\ne BA$$

For example, $${\bf A}=\begin{pmatrix} 1&2\\-1&3\end{pmatrix}, \qquad {\bf B}=\begin{pmatrix} 2&1\\0&1\end{pmatrix}$$ $${\bf AB}=\begin{pmatrix} 2&3\\-2&2\end{pmatrix}, \qquad {\bf BA}=\begin{pmatrix} 1&7\\-1&3\end{pmatrix}$$

**Transpose**: The transpose of the $m\times n$ matrix $\bf A$ is the $n\times m$ matrix ${\bf A}^T$ (also written ${\bf A}'$) obtained by interchanging the rows and columns of $\bf A$.

For example,

${\bf A}=\begin{pmatrix} 4&-2&3\\0&5&-1\end{pmatrix}, \qquad {\bf A}^T=\begin{pmatrix} 4&0\\-2&5\\3&-1 \end{pmatrix}$

${\bf B}=\begin{pmatrix} 2\\-1\\3 \end{pmatrix}, \qquad {\bf B}^T=\begin{pmatrix} 2&-1&3\end{pmatrix}$

The following rules apply for transposed matrices:

```{=tex}
\begin{enumerate}
        \item $({\bf A+B})^T = {\bf A}^T+{\bf B}^T$
        \item $({\bf A}^T)^T={\bf A}$
        \item $(s{\bf A})^T = s{\bf A}^T$
        \item $({\bf AB})^T = {\bf B}^T{\bf A}^T$; and by induction $({\bf ABC})^T = {\bf C}^T{\bf B}^T{\bf A}^T$
\end{enumerate}
```
Example of $({\bf AB})^T = {\bf B}^T{\bf A}^T$: $${\bf A}=\begin{pmatrix} 1&3&2\\2&-1&3\end{pmatrix}, \qquad {\bf B}=\begin{pmatrix} 0&1\\2&2\\3&-1\end{pmatrix}$$ $$ ({\bf AB})^T = \left[ \begin{pmatrix} 1&3&2\\2&-1&3\end{pmatrix} \begin{pmatrix} 0&1\\2&2\\3&-1\end{pmatrix} \right]^T = \begin{pmatrix} 12&7\\5&-3 \end{pmatrix}$$ $$ {\bf B}^T{\bf A}^T= \begin{pmatrix} 0&2&3\\1&2&-1 \end{pmatrix}  \begin{pmatrix} 1&2\\3&-1\\2&3 \end{pmatrix} = \begin{pmatrix} 12&7\\5&-3 \end{pmatrix}$$

::: {#exr-matrixmulti1}
## Matrix Multiplication

Let $$A =  \begin{pmatrix} 2&0&-1&1\\1&2&0&1 \end{pmatrix}$$

$$B = \begin{pmatrix} 1&5&-7\\1&1&0\\0&-1&1\\2&0&0\end{pmatrix} $$

$$C =  \begin{pmatrix} 3&2&-1\\0&4&6 \end{pmatrix}$$

Calculate the following:

1.  $$AB$$
2.  $$BA$$
3.  $$(BC)^T$$
4.  $$BC^T$$
:::

## Systems of Linear Equations

**Linear Equation**: $a_1 x_1 + a_2 x_2 + \cdots + a_n x_n = b$

$a_i$ are parameters or coefficients. $x_i$ are variables or unknowns.

Linear because only one variable per term and degree is at most 1.

We are often interested in solving linear systems like

$$\begin{matrix}
            x  & - & 3y & = & -3\\
            2x & + &  y & = &  8
            \end{matrix}$$

```{=tex}
\begin{comment}
        \parbox[t]{1in}{\includegraphics[angle=270, width = 1in]{linsys.eps}}
\end{comment}
```
More generally, we might have a system of $m$ equations in $n$ unknowns

$$\begin{matrix}
            a_{11}x_1  & + & a_{12}x_2 & + & \cdots & + & a_{1n}x_n & = & b_1\\
            a_{21}x_1  & + & a_{22}x_2 & + & \cdots & + & a_{2n}x_n & = & b_2\\
            \vdots     &   &     &   & \vdots &   &     & \vdots & \\
            a_{m1}x_1  & + & a_{m2}x_2 & + & \cdots & + & a_{mn}x_n & = & b_m
            \end{matrix}$$

A **solution** to a linear system of $m$ equations in $n$ unknowns is a set of $n$ numbers $x_1, x_2, \cdots, x_n$ that satisfy each of the $m$ equations.

Example: $x=3$ and $y=2$ is the solution to the above $2\times 2$ linear system. If you graph the two lines, you will find that they intersect at $(3,2)$.

Does a linear system have one, no, or multiple solutions? For a system of 2 equations with 2 unknowns (i.e., two lines): \_\
**One solution:** The lines intersect at exactly one point.

**No solution:** The lines are parallel.

**Infinite solutions:** The lines coincide.

Methods to solve linear systems:

1.  Substitution
2.  Elimination of variables
3.  Matrix methods

::: {#exr-lineareq}
## Linear Equations

Provide a system of 2 equations with 2 unknowns that has

1.  one solution

2.  no solution

3.  infinite solutions
:::

## Systems of Equations as Matrices

Matrices provide an easy and efficient way to represent linear systems such as $$\begin{matrix}
        a_{11}x_1  & + & a_{12}x_2 & + & \cdots & + & a_{1n}x_n & = & b_1\\
        a_{21}x_1  & + & a_{22}x_2 & + & \cdots & + & a_{2n}x_n & = & b_2\\
        \vdots     &   &     &   & \vdots &   &     & \vdots & \\
        a_{m1}x_1  & + & a_{m2}x_2 & + & \cdots & + & a_{mn}x_n & = & b_m
        \end{matrix}$$

as $${\bf A x = b}$$ where

The $m \times n$ \textbf{coefficient matrix} ${\bf A}$ is an array of $m n$ real numbers arranged in $m$ rows by $n$ columns: $${\bf A}=\begin{pmatrix}
            a_{11} & a_{12} & \cdots & a_{1n} \\
            a_{21} & a_{22} & \cdots & a_{2n} \\
            \vdots &  & \ddots & \vdots \\
            a_{m1} & a_{m2} & \cdots & a_{mn}
            \end{pmatrix}$$

The unknown quantities are represented by the vector ${\bf x}=\begin{pmatrix} x_1\\x_2\\\vdots\\x_n \end{pmatrix}$.

The right hand side of the linear system is represented by the vector ${\bf b}=\begin{pmatrix} b_1\\b_2\\\vdots\\b_m \end{pmatrix}$.

**Augmented Matrix**: When we append $\bf b$ to the coefficient matrix $\bf A$, we get the augmented matrix $\widehat{\bf A}=[\bf A | b]$ $$\begin{pmatrix}
            a_{11} & a_{12} & \cdots & a_{1n} & | & b_1\\
            a_{21} & a_{22} & \cdots & a_{2n} & | & b_2\\
            \vdots &  & \ddots & \vdots & | & \vdots\\
            a_{m1} & a_{m2} & \cdots & a_{mn} & | & b_m
            \end{pmatrix}$$

::: {#exr-augmatrix}
## Augmented Matrix

Create an augmented matrix that represent the following system of equations:

$$2x_1 -7x_2 + 9x_3 -4x_4 = 8$$ $$41x_2 + 9x_3 -5x_6 = 11$$ $$x_1 -15x_2 -11x_5 = 9$$
:::

## Finding Solutions to Augmented Matrices and Systems of Equations

**Row Echelon Form**: Our goal is to translate our augmented matrix or system of equations into row echelon form. This will provide us with the values of the vector **x** which solve the system. We use the row operations to change coefficients in the lower triangle of the augmented matrix to 0. An augmented matrix of the form

$$\begin{pmatrix}
            \fbox{$a'_{11}$}& a'_{12} & a'_{13}& \cdots & a'_{1n} & | & b'_1\\
            0 & \fbox{$a'_{22}$} & a'_{23}& \cdots & a'_{2n} & | & b'_2\\
            0 & 0 & \fbox{$a'_{33}$}& \cdots & a'_{3n} & | & b'_3\\
            0 & 0 &0 & \ddots & \vdots  & | & \vdots \\
            0 & 0 &0 &0 & \fbox{$a'_{mn}$} & | & b'_m
            \end{pmatrix}$$

is said to be in row echelon form --- each row has more leading zeros than the row preceding it.

**Reduced Row Echelon Form**: We can go one step further and put the matrix into reduced row echelon form. Reduced row echelon form makes the value of **x** which solves the system very obvious. For a system of $m$ equations in $m$ unknowns, with no all-zero rows, the reduced row echelon form would be

$$\begin{pmatrix}
            \fbox{$1$}  &  0 &   0 &    0  &   0 & | & b^*_1\\
            0  &  \fbox{$1$} &   0 &    0  &   0 & | & b^*_2\\
            0  &  0 &   \fbox{$1$} &    0  &   0 & | & b^*_3\\
            0  &  0 &   0 &\ddots &   0 & | &\vdots\\
            0  &  0 &   0 &    0  &   \fbox{$1$} & | & b^*_m
            \end{pmatrix}$$

**Gaussian and Gauss-Jordan elimination**: We can conduct elementary row operations to get our augmented matrix into row echelon or reduced row echelon form. The methods of transforming a matrix or system into row echelon and reduced row echelon form are referred to as Gaussian elimination and Gauss-Jordan elimination, respectively.

**Elementary Row Operations**: To do Gaussian and Gauss-Jordan elimination, we use three basic operations to transform the augmented matrix into another augmented matrix that represents an equivalent linear system -- equivalent in the sense that the same values of $x_j$ solve both the original and transformed matrix/system:

**Interchanging Rows**: Suppose we have the augmented matrix $${\widehat{\bf A}}=\begin{pmatrix} a_{11} & a_{12} & | & b_1\\
            a_{21} & a_{22} & | & b_2 
            \end{pmatrix}$$ If we interchange the two rows, we get the augmented matrix $$\begin{pmatrix}
            a_{21} & a_{22} & | & b_2\\
            a_{11} & a_{12} & | & b_1
            \end{pmatrix}$$ which represents a linear system equivalent to that represented by matrix $\widehat{\bf A}$.

**Multiplying by a Constant**: If we multiply the second row of matrix $\widehat{\bf A}$ by a constant $c$, we get the augmented matrix $$\begin{pmatrix}
            a_{11} & a_{12} & | & b_1\\
            c a_{21} & c a_{22} & | & c b_2
            \end{pmatrix}$$ which represents a linear system equivalent to that represented by matrix $\widehat{\bf A}$.

**Adding (subtracting) Rows**: If we add (subtract) the first row of matrix $\widehat{\bf A}$ to the second, we obtain the augmented matrix $$\begin{pmatrix}
            a_{11} & a_{12} & | & b_1\\
            a_{11}+a_{21} & a_{12}+a_{22} & | & b_1+b_2
            \end{pmatrix}$$ which represents a linear system equivalent to that represented by matrix $\widehat{\bf A}$.

::: {#exm-solvesys}
Solve the following system of equations by using elementary row operations:

$\begin{matrix}
            x  & - & 3y & = & -3\\
            2x & + &  y & = &  8
            \end{matrix}$
:::

::: {#exr-solvesys1}
## Solving Systems of Equations

Put the following system of equations into augmented matrix form. Then, using Gaussian or Gauss-Jordan elimination, solve the system of equations by putting the matrix into row echelon or reduced row echelon form.

$$
  1. \begin{cases}
               x + y + 2z = 2\\
               3x - 2y + z = 1\\
               y - z = 3
            \end{cases}
               $$

$$
  2. \begin{cases}
               2x + 3y - z = -8\\
               x + 2y - z = 12\\
             -x -4y + z = -6
            \end{cases}
               $$
:::

## Rank --- and Whether a System Has One, Infinite, or No Solutions

To determine how many solutions exist, we can use information about (1) the number of equations $m$, (2) the number of unknowns $n$, and (3) the **rank** of the matrix representing the linear system.

**Rank**: The maximum number of linearly independent row or column vectors in the matrix. This is equivalent to the number of nonzero rows of a matrix in row echelon form. For any matrix **A**, the row rank always equals column rank, and we refer to this number as the rank of **A**.

For example

$\begin{pmatrix} 1 & 2 & 3 \\
            0 & 4 & 5 \\
            0 & 0 & 6 \end{pmatrix}$

Rank = 3

$\begin{pmatrix} 1 & 2 & 3 \\
0 & 4 & 5 \\
0 & 0 & 0 \end{pmatrix}$

Rank = 2

::: {#exr-rank}
## Rank of Matrices

Find the rank of each matrix below:

(Hint: transform the matrices into row echelon form. Remember that the number of nonzero rows of a matrix in row echelon form is the rank of that matrix)

1.$\begin{pmatrix} 1 & 1 & 2 \\
2 & 1 & 3 \\
1 & 2 & 3 \end{pmatrix}$

\bigskip

2.$\begin{pmatrix} 1 & 3 & 3 & -3 & 3\\
1 & 3 & 1 & 1 & 3 \\
1 & 3 & 2 & -1 & -2 \\
1 & 3 & 0 & 3 & -2 \end{pmatrix}$
:::

Answer to Exercise @exr-rank:

1.  rank is 2

2.  rank is 3

## The Inverse of a Matrix

**Identity Matrix**: The $n\times n$ identity matrix ${\bf I}_n$ is the matrix whose diagonal elements are 1 and all off-diagonal elements are 0. Examples: $$ {\bf I}_2=\begin{pmatrix} 1&0\\0&1 \end{pmatrix}, \qquad {\bf I}_3=\begin{pmatrix} 1&0&0\\ 0&1&0\\ 
            0&0&1 \end{pmatrix}$$

**Inverse Matrix**: An $n\times n$ matrix ${\bf A}$ is **nonsingular** or **invertible** if there exists an $n\times n$ matrix ${\bf A}^{-1}$ such that $${\bf A} {\bf A}^{-1} = {\bf A}^{-1} {\bf A} = {\bf I}_n$$ where ${\bf A}^{-1}$ is the inverse of ${\bf A}$. If there is no such ${\bf A}^{-1}$, then ${\bf A}$ is singular or not invertible.

Example: Let $${\bf A} = \begin{pmatrix} 2&3\\2&2 \end{pmatrix}, \qquad {\bf B}=\begin{pmatrix} -1&\frac{3}{2}\\ 1&-1
        \end{pmatrix}$$ Since $${\bf A} {\bf B} = {\bf B} {\bf A} = {\bf I}_n$$ we conclude that ${\bf B}$ is the inverse, ${\bf A}^{-1}$, of ${\bf A}$ and that ${\bf A}$ is nonsingular.

**Properties of the Inverse**:

-   If the inverse exists, it is unique.

-   If ${\bf A}$ is nonsingular, then ${\bf A}^{-1}$ is nonsingular.

-   $({\bf A}^{-1})^{-1} = {\bf A}$

-   If ${\bf A}$ and ${\bf B}$ are nonsingular, then ${\bf A}{\bf B}$ is nonsingular

-   $({\bf A}{\bf B})^{-1} = {\bf B}^{-1}{\bf A}^{-1}$

-   If ${\bf A}$ is nonsingular, then $({\bf A}^T)^{-1}=({\bf A}^{-1})^T$

**Procedure to Find** ${\bf A}^{-1}$: We know that if ${\bf B}$ is the inverse of ${\bf A}$, then $${\bf A} {\bf B} = {\bf B} {\bf A} = {\bf I}_n$$ Looking only at the first and last parts of this $${\bf A} {\bf B} = {\bf I}_n$$ Solving for ${\bf B}$ is equivalent to solving for $n$ linear systems, where each column of ${\bf B}$ is solved for the corresponding column in ${\bf I}_n$. We can solve the systems simultaneously by augmenting ${\bf A}$ with ${\bf I}_n$ and performing Gauss-Jordan elimination on ${\bf A}$. If Gauss-Jordan elimination on $[{\bf A} | {\bf I}_n]$ results in $[{\bf I}_n | {\bf B} ]$, then ${\bf B}$ is the inverse of ${\bf A}$. Otherwise, ${\bf A}$ is singular.

To summarize: To calculate the inverse of ${\bf A}$

1.  Form the augmented matrix $[ {\bf A} | {\bf I}_n]$

2.  Using elementary row operations, transform the augmented matrix to reduced row echelon form.

3.  The result of step 2 is an augmented matrix $[ {\bf C} | {\bf B} ]$.

    a.  If ${\bf C}={\bf I}_n$, then ${\bf B}={\bf A}^{-1}$.

    b.  If ${\bf C}\ne{\bf I}_n$, then $\bf C$ has a row of zeros. This means ${\bf A}$ is singular and ${\bf A}^{-1}$ does not exist.

::: {#exm-inverse}
Find the inverse of the following matricies:

1.  ${\bf A}=\begin{pmatrix} 1&1&1\\0&2&3\\5&5&1 \end{pmatrix}$
:::

::: {#exr-inverse1}
## Finding the inverse of matrices

Find the inverse of the following matrix:

1.  ${\bf A}=\begin{pmatrix} 1&0&4\\0&2&0\\0&0&1 \end{pmatrix}$
:::

## Linear Systems and Inverses

Let's return to the matrix representation of a linear system

$$\bf{Ax} = \bf{b}$$

If $\bf{A}$ is an $n\times n$ matrix,then $\bf{Ax}=\bf{b}$ is a system of $n$ equations in $n$ unknowns. Suppose $\bf{A}$ is nonsingular. Then $\bf{A}^{-1}$ exists. To solve this system, we can multiply each side by $\bf{A}^{-1}$ and reduce it as follows:

```{=tex}
\begin{eqnarray*} 
\bf{A}^{-1} (\bf{A} \bf{x}) & = & \bf{A}^{-1} \bf{b} \\
(\bf{A}^{-1} \bf{A})\bf{x} & = & \bf{A}^{-1} \bf{b}\\
\bf{I}_n \bf{x}     & = & \bf{A}^{-1} \bf{b}\\
\bf{x} & = & \bf{A}^{-1} \bf{b}
\end{eqnarray*}
```
Hence, given $\bf{A}$ and $\bf{b}$ and given that $\bf{A}$ is nonsingular, then $\bf{x} = \bf{A}^{-1} \bf{b}$ is a unique solution to this system.

::: {#exr-invlinsys}
## Solve linear system using inverses

Use the inverse matrix to solve the following linear system:

```{=tex}
\begin{align*} 
-3x + 4y &= 5 \\
2x - y &= -10
\end{align*}
```
***Hint: the linear system above can be written in the matrix form***

$\textbf{A}\textbf{z} = \textbf{b}$

given $$\textbf{A} = \begin{pmatrix} -3&4\\2&-1 \end{pmatrix},$$

$$\textbf{z} = \begin{pmatrix} x\\y \end{pmatrix},$$ and $$\textbf{b} = \begin{pmatrix} 5\\-10 \end{pmatrix}$$
:::

## Determinants

**Singularity**: Determinants can be used to determine whether a square matrix is nonsingular.

A square matrix is nonsingular if and only if its determinant is not zero.

Determinant of a $1 \times 1$ matrix, **A**, equals $a_{11}$

Determinant of a $2 \times 2$ matrix, **A**, $\begin{vmatrix} a_{11}&a_{12}\\
            a_{21}&a_{22} \end{vmatrix}$:

```{=tex}
\begin{eqnarray*}
\det({\bf A}) &=& |{\bf A}|\\
            &=& a_{11}|a_{22}| - a_{12}|a_{21}|\\
            &=& a_{11}a_{22} - a_{12}a_{21}
\end{eqnarray*}
```
We can extend the second to last equation above to get the definition of the determinant of a $3 \times 3$ matrix:

```{=tex}
\begin{eqnarray*}
            \begin{vmatrix} a_{11}&a_{12}&a_{13}\\  a_{21} & a_{22}&a_{23}\\ a_{31}&a_{32}&a_{33} \end{vmatrix} 
                &=& 
                a_{11} \begin{vmatrix} a_{22}&a_{23}\\ a_{32}&a_{33} \end{vmatrix}
                - a_{12} \begin{vmatrix} a_{21}&a_{23}\\ a_{31}&a_{33} \end{vmatrix}
                + a_{13} \begin{vmatrix} a_{21}&a_{22}\\ a_{31}&a_{32} 
                \end{vmatrix}\\
                &=& a_{11}(a_{22}a_{33} - a_{23}a_{32}) - a_{12}(a_{21}a_{33} - a_{23}a_{31}) + a_{13}(a_{21}a_{32} - a_{22}a_{31})
\end{eqnarray*}
```
Let's extend this now to any $n\times n$ matrix. Let's define ${\bf A}_{ij}$ as the $(n-1)\times (n-1)$ submatrix of ${\bf A}$ obtained by deleting row $i$ and column $j$. Let the $(i,j)$th **minor** of ${\bf A}$ be the determinant of ${\bf A}_{ij}$: $$M_{ij}=|{\bf A}_{ij}|$$ Then for any $n\times n$ matrix ${\bf A}$ $$|{\bf A}|= a_{11}M_{11} - a_{12}M_{12} + \cdots + (-1)^{n+1} a_{1n} M_{1n}$$

For example, in figuring out whether the following matrix has an inverse? $${\bf A}=\begin{pmatrix} 1&1&1\\0&2&3\\5&5&1 \end{pmatrix}$$ 1. Calculate its determinant. \begin{eqnarray}
                &=& 1(2-15) - 1(0-15) + 1(0-10) \nonumber\\
                &=& -13+15-10 \nonumber\\
                &=& -8\nonumber
\end{eqnarray} 2. Since $|{\bf A}|\ne 0$, we conclude that ${\bf A}$ has an inverse.

::: {#exr-determinants}
## Determinants and Inverses

Determine whether the following matrices are nonsingular:

$$1. \begin{pmatrix}
            1 & 0 & 1\\
            2 & 1 & 2\\
            1 & 0 & -1
        \end{pmatrix}$$

$$2. \begin{pmatrix}
            2 & 1 & 2\\
            1 & 0 & 1\\
            4 & 1 & 4
        \end{pmatrix}$$
:::

## Getting Inverse of a Matrix using its Determinant

Thus far, we have a number of algorithms to

1.  Find the solution of a linear system,
2.  Find the inverse of a matrix

but these remain just that --- algorithms. At this point, we have no way of telling how the solutions $x_j$ change as the parameters $a_{ij}$ and $b_i$ change, except by changing the values and "rerunning" the algorithms.

With determinants, we can provide an explicit formula for the inverse and therefore provide an explicit formula for the solution of an $n\times n$ linear system.

Hence, we can examine how changes in the parameters and $b_i$ affect the solutions $x_j$.

**Determinant Formula for the Inverse of a** $2 \times 2$:

The determinant of a $2 \times 2$ matrix **A** $\begin{pmatrix} a & b\\ c & d\\ \end{pmatrix}$ is defined as: $$\frac{1}{\det({\bf A})} \begin{pmatrix}
            d & -b\\
            -c & a\\
        \end{pmatrix}$$

For example, Let's calculate the inverse of matrix A from Exercise @exr-invlinsys using the determinant formula.

Recall,

$$A = \begin{pmatrix}
            -3 & 4\\
            2 & -1\\
        \end{pmatrix}$$

$$\det({\bf A}) = (-3)(-1) - (4)(2) = 3 - 8  = -5$$

$$\frac{1}{\det({\bf A})} \begin{pmatrix}
            -1 & -4\\
            -2 & -3\\
        \end{pmatrix}$$

$$\frac{1}{-5} \begin{pmatrix}
            -1 & -4\\
            -2 & -3\\
        \end{pmatrix}$$

$$ \begin{pmatrix}
            \frac{1}{5} & \frac{4}{5}\\
            \frac{2}{5} & \frac{3}{5}\\
        \end{pmatrix}$$

::: {#exr-calcinverse}
## Calculate Inverse using Determinant Formula

Caculate the inverse of A

$$A = \begin{pmatrix}
            3 & 5\\
            -7 & 2\\
        \end{pmatrix}$$
:::

## Answers to Examples and Exercises {.unnumbered}

Answer to Example @exm-vectors:

1.  $\begin{pmatrix} -1 &-3&-3 \end{pmatrix}$
2.  6 + 4 + 10 = 20

Answer to Exercise @exr-vectors1:

1.  $\begin{pmatrix} -2 &4&-7&-5 \end{pmatrix}$
2.  $\begin{pmatrix} 2 &26&-14&4&30 \end{pmatrix}$
3.  63 -3 -10 + 24 = 74
4.  undefined

Answer to Example @exm-linearindep:

1.  yes
2.  no

Answer to Exercise @exr-linearindep1:

1.  yes
2.  no ($-v_1 -v_2 + v_3  = 0$)

Answer to Example @exm-matrixaddition:

${\bf A+B}=\begin{pmatrix} 2 & 4 & 4 \\ 6 & 6 & 8 \end{pmatrix}$

Answer to Example @exm-scalarmulti:

$s {\bf A} = \begin{pmatrix} 2 & 4 & 6 \\ 8 & 10 & 12 \end{pmatrix}$

Answer to Example @exm-matrixmulti:

1.  $\begin{pmatrix} aA+bC&aB+bD\\cA+dC&cB+dD\\eA+fC&eB+fD \end{pmatrix}$

2.  $\begin{pmatrix} 1(-2)+2(4)-1(2)&1(5)+2(-3)-1(1)\\
                 3(-2)+1(4)+4(2)&3(5)+1(-3)+4(1)\end{pmatrix} =
             \begin{pmatrix} 4&-2\\6&16\end{pmatrix}$

Answer to Exercise @exr-matrixmulti1:

1.  $AB = \begin{pmatrix} 4 & 11 & -15 \\ 5 & 7 & -7 \end{pmatrix}$

2.  $BA =$ undefined

3.  $(BC)^T =$ undefined

4.  $BC^T = \begin{pmatrix} 1&5&-7\\1&1&0\\0&-1&1\\2&0&0\end{pmatrix}\begin{pmatrix} 3&0\\2&4\\-1&6 \end{pmatrix} =\begin{pmatrix}20 & -22 \\ 5 & 4 \\ -3 &2 \\6 & 0\end{pmatrix}$

Answer to Exercise @exr-lineareq:

There are many answers to this. Some possible simple ones are as follows:

1.  One solution: $$\begin{matrix}
             -x  & + & y & = & 0\\
             x & + &  y & = &  2
             \end{matrix}$$

2.  No solution: $$\begin{matrix}
            -x  & + & y & = & 0\\
            x & - &  y & = &  2
            \end{matrix}$$

3.  Infinite solutions: $$\begin{matrix}
             -x  & + & y & = & 0\\
             2x & - &  2y & = &  0
             \end{matrix}$$

Answer to Exercise @exr-augmatrix:

$\begin{pmatrix}
            2 & -7 & 9 & -4 & 0 & 0| & 8\\
            0 & 41 & 9 & 0 & 0 & 5 | & 11\\
            1 & -15 & 0 & 0 & -11 & 0 | & 9
            \end{pmatrix}$

Answer to Example @exm-solvesys:

$$\begin{matrix}
            x  & - & 3y & = & -3\\
            2x & + &  y & = &  8
            \end{matrix}$$

$$\begin{matrix}
            x  & - & 3y & = & -3\\
               &   & 7y & = & 14\\          
            \end{matrix}$$

$$\begin{matrix}
            x  & - & 3y & = & -3\\
               &   & y & = & 2\\            
            \end{matrix}$$

$$\begin{matrix}
            x & = & 3\\
            y & = & 2\\         
            \end{matrix}$$

Answer to Exercise @exr-solvesys1:

1.  x = 2, y = 2, z = -1

2.  x = -17, y = -3, z = -35

Answer to Exercise @exr-rank:

1.  rank is 2

2.  rank is 3

Answer to Example @exm-inverse:

$\left(\begin{array}{ccc|ccc}
    1&1&1&1&0&0\\
    0&2&3&0&1&0\\
    5&5&1&0&0&1
\end{array} \right)$

$\left(\begin{array}{ccc|ccc}
    1&1&1 &1 &0&0\\
    0&2&3 &0 &1&0\\
    0&0&-4&-5&0&1
\end{array} \right)$

$\left(\begin{array}{ccc|ccc}
    1&1&1&1  &0&0\\
    0&2&3&0  &1&0\\
    0&0&1&5/4&0&-1/4
\end{array} \right)$

$\left(\begin{array}{ccc|ccc}
    1&1&0&-1/4 &0&1/4\\
    0&2&0&-15/4&1&3/4\\
    0&0&1&5/4  &0&-1/4
\end{array} \right)$

$\left(\begin{array}{ccc|ccc}
    1&1&0&-1/4 &0  &1/4\\
    0&1&0&-15/8&1/2&3/8\\
    0&0&1&5/4  &0 &-1/4
\end{array} \right)$

$\left(\begin{array}{ccc|ccc}
    1&0&0&13/8 &-1/2&-1/8\\
    0&1&0&-15/8&1/2 &3/8\\
    0&0&1&5/4  &0   &-1/4
\end{array} \right)$

${\bf A}^{-1} = \left(\begin{array}{ccc}
    13/8 &-1/2&-1/8\\
    -15/8&1/2 &3/8\\
    5/4  &0   &-1/4
\end{array} \right)$

Answer to Exercise @exr-inverse1:

1.  ${\bf A}^{-1}=\begin{pmatrix} 1&0&-4\\0&\frac{1}{2}&0\\0&0&1 \end{pmatrix}$

Answer to Exercise @exr-invlinsys:

$\textbf{z} = \bf{A}^{-1} \bf{b} = \begin{pmatrix}
    1/5 &4/5\\
    2/5&3/5
\end{pmatrix} \begin{pmatrix}
    5 \\
    -10
\end{pmatrix}= \begin{pmatrix}
    -7 \\
    -4
\end{pmatrix} = \begin{pmatrix}
    x \\
    y
\end{pmatrix}$

Answer to Exercise @exr-determinants:

1.  nonsingular

2.  singular

Answer to Exercise @exr-calcinverse:

$\begin{pmatrix}
            \frac{2}{41} & \frac{-5}{41}\\
            \frac{7}{41} & \frac{3}{41}\\
        \end{pmatrix}$