Last updated: Mon Mar 31 08:14:32 2025Introduction to Linear Algebra
What is Linear Algebra?
Linear Algebra is a branch of mathematics that focuses on the study of linear spaces, linear transformations, and systems of linear equations. At its core, linear algebra provides the tools for analyzing and solving problems related to these mathematical objects, making it an essential area of study in various fields, including computer science, engineering, physics, and economics.
Matrix calculus
General form of a matrix
Suppose we have two sets of indices \(i\in\{1,\ldots,n\}\) and \(j\in\{1,\ldots,m\}\). Let us define a mapping that assigns a number \(a_{ij}\) to each pair of indices \((i,j)\). This mapping is called a matrix and is denoted by \(\mathbf{A}\).
Because pair of integers can be drawn as a two-dimensional grid, we can represent a matrix as a rectangular array of numbers. The size of the matrix is given by the number of rows and columns. For example, a matrix of size \(n \times m\) has \(n\) rows and \(m\) columns:
\[ \mathbf{A}= [a_{ij}]_{n \times m}= \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1m} \\ a_{21} & a_{22} & \cdots & a_{2m} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nm} \end{bmatrix} \]
Where \(a_{ij}\) represents the element in the \(i\)-th row and \(j\)-th column.
2x2 Matrix
A 2x2 matrix (2 rows and 2 columns) looks like this:
\[ \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix} \]
Where \(a_{ij}\) represents the element in the \(i\)-th row and \(j\)-th column.
3x5 Matrix
A 3x5 matrix (3 rows and 5 columns) looks like this:
\[ \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} & a_{15} \\ a_{21} & a_{22} & a_{23} & a_{24} & a_{25} \\ a_{31} & a_{32} & a_{33} & a_{34} & a_{35} \end{bmatrix} \]
Where \(a_{ij}\) represents the element in the \(i\)-th row and \(j\)-th column.
Special Matrices
Square Matrix
A square matrix has the same number of rows and columns. For example, a \(3 \times 3\) matrix is a square matrix because it has 3 rows and 3 columns.
Diagonal Matrix
A diagonal matrix is a square matrix in which all the elements outside the main diagonal are zero. The main diagonal is the set of elements where the row and column indices are equal. For example, a \(3 \times 3\) diagonal matrix looks like this:
\[ \begin{bmatrix} a_{11} & 0 & 0 \\ 0 & a_{22} & 0 \\ 0 & 0 & a_{33} \end{bmatrix} \]
Identity Matrix
The identity matrix is a special diagonal matrix in which all the elements on the main diagonal are equal to 1. We denote the identity matrix by \(\mathbf{I}\) or \(\mathbf{I}_n\) for an \(n \times n\) matrix. For example, the \(3 \times 3\) identity matrix is:
\[ \mathbf{I}_3 = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \]
Zero Matrix
The zero matrix is a matrix in which all the elements are zero. We denote the zero matrix by \(\mathbf{0}\) or \(\mathbf{0}_{n \times m}\) for an \(n \times m\) matrix.
Symmetric Matrix
A symmetric matrix is a square matrix that is equal to its transpose. In other words, the element in the \(i\)-th row and \(j\)-th column is equal to the element in the \(j\)-th row and \(i\)-th column. For example, a \(3 \times 3\) symmetric matrix looks like this:
\[ \begin{bmatrix} a_{11} & \color{red}{b} & \color{green}{c} \\ \color{red}{b} & a_{22} & \color{blue}{d} \\ \color{green}{c} & \color{blue}{d} & a_{33} \end{bmatrix} \]
Triangular Matrix
A triangular matrix is a square matrix in which all the elements above or below the main diagonal are zero. A matrix is called upper triangular if all the elements below the main diagonal are zero, and lower triangular if all the elements above the main diagonal are zero. For example, a \(3 \times 3\) upper triangular matrix looks like this:
\[ \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ \color{red}{0} & a_{22} & a_{23} \\ \color{red}{0} & \color{red}{0} & a_{33} \end{bmatrix} \]
Matrix Operations
Lecture Notebooks: LA_Matrix_basic_operations
Addition
The addition of two matrices is defined only for matrices of the same size. The sum of two matrices is obtained by adding the elements of the matrices with the same row and column indices.
Let’s consider two matrices \(\mathbf{A}\) and \(\mathbf{B}\) of the size \(2 \times 2\):
The sum of matrices \(\mathbf{A}\) and \(\mathbf{B}\) is given by:
\[ \mathbf{A} + \mathbf{B} = \begin{bmatrix} \color{blue}{a_{11}} & \color{blue}{a_{12}} \\ \color{blue}{a_{21}} & \color{blue}{a_{22}} \end{bmatrix} + \begin{bmatrix} \color{red}{b_{11}} & \color{red}{b_{12}} \\ \color{red}{b_{21}} & \color{red}{b_{22}} \end{bmatrix} = \begin{bmatrix} \color{blue}{a_{11}} + \color{red}{b_{11}} & \color{blue}{a_{12}} + \color{red}{b_{12}} \\ \color{blue}{a_{21}} + \color{red}{b_{21}} & \color{blue}{a_{22}} + \color{red}{b_{22}} \end{bmatrix} \]
Multiplication by scalar
Let’s consider a matrix \(\mathbf{A}\) of the size \(2 \times 2\) and a scalar (number) \(c\):
\[ \mathbf{A} = \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix} \]
The product of the scalar \(c\) and the matrix \(\mathbf{A}\) is given by:
\[ \textcolor{red}{c} \cdot \mathbf{A} = \textcolor{red}{c} \cdot \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix} = \begin{bmatrix} \textcolor{red}{c} \cdot a_{11} & \textcolor{red}{c} \cdot a_{12} \\ \textcolor{red}{c} \cdot a_{21} & \textcolor{red}{c} \cdot a_{22} \end{bmatrix} \]
Matrix Multiplication
General Form
Let’s consider two matrices \(\mathbf{A}\) of size \({\color{red}n} \times {\color{green}m}\) and \(\mathbf{B}\) of size \({\color{green}m} \times {\color{blue}p}\). The product of matrices \(\mathbf{A}\) and \(\mathbf{B}\) is a matrix \(\mathbf{C}\) of size \({\color{red}n} \times {\color{blue}p}\).
Each element \(c_{ij}\) of the resulting matrix \(\mathbf{C}\) is calculated as:
\[ c_{\color{red}{i}\color{blue}{j}} = \sum_{{\color{green}k}=1}^{m} a_{\color{red}{i}\color{green}{k}} b_{\color{green}{k}\color{blue}{j}} = a_{\color{red}{i}{\color{green}1}} b_{{\color{green}1}{\color{blue}j}} + a_{\color{red}{i}{\color{green}2}} b_{{\color{green}2}{\color{blue}j}} + \ldots + a_{\color{red}{i}{\color{green}m}}b_{{\color{green}m}{\color{blue}j}} \]
for \(i = 1, 2, \ldots, n\) and \(j = 1, 2, \ldots, p\).
In other words, the element in the \(i\)-th row and \(j\)-th column of \(\mathbf{C}\) is the sum of the products of the elements from the \(i\)-th row of \(\mathbf{A}\) and the \(j\)-th column of \(\mathbf{B}\).
2x2 Matrices
Let’s consider two matrices \(\mathbf{A}\) and \(\mathbf{B}\) of the size \(2 \times 2\):
\[ \mathbf{A} = \begin{bmatrix} \color{red}{a_{11}} & \color{red}{a_{12}} \\ \color{blue}{a_{21}} & \color{blue}{a_{22}} \end{bmatrix} \]
\[ \mathbf{B} = \begin{bmatrix} \color{green}{b_{11}} & \color{orange}{b_{12}} \\ \color{green}{b_{21}} & \color{orange}{b_{22}} \end{bmatrix} \]
The product of matrices \(\mathbf{A}\) and \(\mathbf{B}\) is given by:
\[ \mathbf{A} \cdot \mathbf{B} = \begin{bmatrix} \color{red}{a_{11}} & \color{red}{a_{12}} \\ \color{blue}{a_{21}} & \color{blue}{a_{22}} \end{bmatrix} \cdot \begin{bmatrix} \color{green}{b_{11}} & \color{orange}{b_{12}} \\ \color{green}{b_{21}} & \color{orange}{b_{22}} \end{bmatrix} = \begin{bmatrix} \color{red}{a_{11}}\color{green}{b_{11}} + \color{red}{a_{12}}\color{green}{b_{21}} & \color{red}{a_{11}}\color{orange}{b_{12}} + \color{red}{a_{12}}\color{orange}{b_{22}} \\ \color{blue}{a_{21}}\color{green}{b_{11}} + \color{blue}{a_{22}}\color{green}{b_{21}} & \color{blue}{a_{21}}\color{orange}{b_{12}} + \color{blue}{a_{22}}\color{orange}{b_{22}} \end{bmatrix} \]
Non-Commutativity of Matrix Multiplication
Consider a \(1 \times 2\) row matrix \(\mathbf{A}\) and a \(2 \times 1\) column matrix \(\mathbf{B}\):
\[ \mathbf{A} = \begin{bmatrix} a_{11} & a_{12} \end{bmatrix} \]
\[ \mathbf{B} = \begin{bmatrix} b_{11} \\ b_{21} \end{bmatrix} \]
The product \(\mathbf{A} \cdot \mathbf{B}\) is:
\[ \mathbf{A} \cdot \mathbf{B} = \begin{bmatrix} \color{blue}{a_{11}} & \color{blue}{a_{12}} \end{bmatrix} \cdot \begin{bmatrix} \color{red}{b_{11}} \\ \color{red}{b_{21}} \end{bmatrix} = [\color{blue}{a_{11}}\color{red}{b_{11}} + \color{blue}{a_{12}}{\color{red}{b_{21}}}] \]
Now, let’s consider the product \(\mathbf{B} \cdot \mathbf{A}\):
\[ \mathbf{B} \cdot \mathbf{A} = \begin{bmatrix} \color{red}{b_{11}} \\ \color{red}{b_{21}} \end{bmatrix} \cdot \begin{bmatrix} \color{blue}{a_{11}} & \color{blue}{a_{12}} \end{bmatrix} = \begin{bmatrix} \color{red}{b_{11}}\color{blue}{a_{11}} & \color{red}{b_{11}}\color{blue}{a_{12}} \\ \color{red}{b_{21}}\color{blue}{a_{11}} & \color{red}{b_{21}}\color{blue}{a_{12}} \end{bmatrix} \]
As we can see, \(\mathbf{A} \cdot \mathbf{B}\) is a scalar, while \(\mathbf{B} \cdot \mathbf{A}\) is a \(2 \times 2\) matrix.
Therefore, \(\mathbf{A} \cdot \mathbf{B} \neq \mathbf{B} \cdot \mathbf{A}\), demonstrating that matrix multiplication is not commutative.
Transpose
The transpose of a matrix \(\mathbf{A}\) is denoted by \(\mathbf{A}^T\) and is obtained by swapping the rows and columns of the matrix. For example, if \(\mathbf{A}\) is a \(2 \times 3\) matrix:
Examples
\[ \begin{bmatrix} {\color{red} 1} & {\color{red} 2} & {\color{red} 3} \\ {\color{blue} 4} & {\color{blue} 5} & {\color{blue} 6} \end{bmatrix}^T = \begin{bmatrix} {\color{red} 1} & {\color{blue} 4} \\ {\color{red} 2} & {\color{blue} 5} \\ {\color{red} 3} & {\color{blue} 6} \end{bmatrix} \]
\[ \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}^T = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix} \]
Determinants
Determinants are a scalar value that can be computed from the elements of a square matrix and provide important information about the matrix. The determinant of a matrix is denoted by \(\text{det}(\mathbf{A})\) or \(|\mathbf{A}|\).
2x2 Matrices
For a \(2 \times 2\) matrix \(\mathbf{A}\):
\[ \det(\mathbf{A}) = \left| \begin{matrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{matrix} \right| = a_{11}a_{22} - a_{12}a_{21} \]
3x3 Matrices
For a \(3 \times 3\) matrix \(\mathbf{A}\):
\[ \det(\mathbf{A}) = \left| \begin{matrix} \color{red}{a_{11}} & \color{green}{a_{12}} & \color{blue}{a_{13}} \\ \color{red}{a_{21}} & \color{green}{a_{22}} & \color{blue}{a_{23}} \\ \color{red}{a_{31}} & \color{green}{a_{32}} & \color{blue}{a_{33}} \end{matrix} \right| = {\color{red}{a_{11}}}({\color{green}{a_{22}}}{\color{blue}{a_{33}}} - {\color{green}{a_{23}}}{\color{blue}{a_{32}}}) - {\color{red}{a_{21}}}({\color{green}{a_{12}}}{\color{blue}{a_{33}}} - {\color{green}{a_{13}}}{\color{blue}{a_{32}}}) + {\color{red}{a_{31}}}({\color{green}{a_{12}}}{\color{blue}{a_{23}}} - {\color{green}{a_{13}}}{\color{blue}{a_{22}}}) \]
To visualize the calculation of the determinant of a \(3 \times 3\) matrix, we can use the rule of Sarrus:
Minors and Cofactors
Minors
A minor of a matrix is the determinant of a sub-matrix obtained by removing one or more rows and columns from the original matrix. For an \(n \times n\) matrix \(\mathbf{A}\), the minor \(M_{ij}\) is the determinant of \((n-1) \times (n-1)\) submatrix obtained by removing the \(i\)-th row and \(j\)-th column from \(\mathbf{A}\).
Cofactors
A cofactor \(C_{ij}\) is the minor of a matrix with an additional sign factor
\[ C_{ij} = (-1)^{i+j} M_{ij} \]
where \((-1)^{i+j}\) is the sign factor that depends on the row and column indices.
For example, consider the following \(3 \times 3\) matrix \(\mathbf{A}\):
\[ \mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix} \]
The minor \(M_{12}\) is determined by removing the first row and second column from \(\mathbf{A}\):
\[ M_{12} = \begin{vmatrix} \times & \times & \times \\ 4 &\times & 6 \\ 7 & \times & 9 \end{vmatrix} = \begin{vmatrix} 4 & 6 \\ 7 & 9 \end{vmatrix} = 4 \times 9 - 6 \times 7 = 36 - 42 = -6 \]
Then cofactor \(C_{12}\) is the minor \(M_{12}\) with the sign factor \((-1)^{1+2} = -1\):
\[ C_{12} = -1 \cdot M_{12} = 6 \]
Analogously, the minor \(M_{22}\)
\[ M_{22} = \begin{vmatrix} 1 & \times & 3 \\ \times & \times & \times \\ 7 & \times & 9 \end{vmatrix} = \begin{vmatrix} 1 & 3 \\ 7 & 9 \end{vmatrix} = 1 \times 9 - 3 \times 7 = 9 - 21 = -12 \]
The cofactor \(C_{22}\) is the minor \(M_{22}\) with the sign factor \((-1)^{2+2} = 1\):
\[ C_{22} = 1 \cdot A_{22} = -12 \]
Number of minors and cofactors in a matrix is equal to the number of elements in the matrix. In case of a \(3 \times 3\) matrix, there are 9 minors and 9 cofactors.
Laplace Expansion - Determinants of Larger Matrices
The Laplace expansion, also known as cofactor expansion, is a method for calculating the determinant of an \(n \times n\) matrix. This method involves expanding the determinant along a row or a column.
For an \(n \times n\) matrix \(\mathbf{A}\), the determinant can be calculated by expanding along the \(i\)-th row as follows:
Expansion along the rows \[ \begin{align*} \det(\mathbf{A}) &= \sum_{j=1}^{n} (-1)^{i+j} a_{1j} M_{1j}= \sum_{j=1}^{n} a_{1j} C_{1j} & \text{first row}\\ &= \sum_{j=1}^{n} (-1)^{i+j} a_{2j} M_{2j}= \sum_{j=1}^{n} a_{2j} C_{2j} & \text{second row}\\ &\qquad \vdots & \\ &= \sum_{j=1}^{n} (-1)^{i+j} a_{nj} M_{nj}= \sum_{j=1}^{n} a_{nj} C_{nj} & \text{n-th row} \end{align*} \]
where \(a_{ij}\) is the element of the matrix in the \(i\)-th row and \(j\)-th column, and \(M_{ij}\) is the minor of the element \(a_{ij}\) and \(C_{ij}\) is the cofactor of the element \(a_{ij}\).
Expansion along the columns
Similarly, the determinant can be calculated by expanding along the \(j\)-th column:
\[ \begin{align*} \det(\mathbf{A}) &= \sum_{i=1}^{n} (-1)^{i+j} a_{i1} M_{i1} = \sum_{i=1}^{n} a_{i1} C_{i1} & \text{first column} \\ &= \sum_{i=1}^{n} (-1)^{i+j} a_{i2} M_{i2} = \sum_{i=1}^{n} a_{i2} C_{i2} & \text{second column} \\ &\qquad \vdots \\ &= \sum_{i=1}^{n} (-1)^{i+j} a_{in} M_{in} = \sum_{i=1}^{n} a_{in} C_{in} & \text{n-th column} \end{align*} \]
where \(a_{ij}\) is the element of the matrix in the \(i\)-th row and \(j\)-th column, and \(M_{ij}\) is the minor of the element \(a_{ij}\) and \(C_{ij}\) is the cofactor of the element \(a_{ij}\).
Note
For a \(3 \times 3\) matrix, there are \(2 \times 3 = 6\) possible ways to expand the determinant using Laplace expansion (3 rows and 3 columns). It is generally advantageous to choose the row or column that contains the most zeros, as this simplifies the calculation.
Example
Consider the following \(4 \times 4\) matrix \(\mathbf{A}\):
\[ \mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\ a_{21} & a_{22} & a_{23} & a_{24} \\ a_{31} & a_{32} & a_{33} & a_{34} \\ a_{41} & a_{42} & a_{43} & a_{44} \end{bmatrix} \]
The determinant of \(\mathbf{A}\) can be calculated by expanding along the first row:
\[ \det(\mathbf{A}) = a_{11} M_{11} - a_{12} M_{12} + a_{13} M_{13} - a_{14} M_{14} \]
where:
\[ M_{11} = \begin{vmatrix} a_{22} & a_{23} & a_{24} \\ a_{32} & a_{33} & a_{34} \\ a_{42} & a_{43} & a_{44} \end{vmatrix} \quad\text{and}\quad M_{12} = \begin{vmatrix} a_{21} & a_{23} & a_{24} \\ a_{31} & a_{33} & a_{34} \\ a_{41} & a_{43} & a_{44} \end{vmatrix} \]
\[ M_{13} = \begin{vmatrix} a_{21} & a_{22} & a_{24} \\ a_{31} & a_{32} & a_{34} \\ a_{41} & a_{42} & a_{44} \end{vmatrix} \quad\text{and}\quad M_{14} = \begin{vmatrix} a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \\ a_{41} & a_{42} & a_{43} \end{vmatrix} \]
Explicitly
\[ \mathbf{A} = \begin{bmatrix} 1&2&3&4\\ 0&1&0&0\\ 2&5&1&4\\ 4&3&2&1 \end{bmatrix} \]
We will calculate the determinant of the matrix by expanding along the second row:
\[ \det(\mathbf{A}) = -0 \cdot M_{21} + 1 \cdot M_{22} - 0 \cdot M_{23} + 0 \cdot M_{24} \]
where:
\[ M_{22}= \begin{vmatrix} 1 & 3 & 4 \\ 2 & 1 & 4 \\ 4 & 2 & 1 \end{vmatrix} = 35 \]
so
\[ \det(\mathbf{A}) = 35 \]
Properties
Determinants have several properties that can simplify their calculation:
- Linearity: The determinant is a linear function of the rows (or columns) of the matrix. This means that if you multiply a row by a scalar, the determinant is also multiplied by that scalar.
\[ \begin{vmatrix} a_{11} & {\color{red}{a_{12}}} + {\color{green}{k}} {\color{blue}{b_{12}}} & a_{13} \\ a_{21} & {\color{red}{a_{22}}} + {\color{green}{k}} {\color{blue}{b_{22}}} & a_{23} \\ a_{31} & {\color{red}{a_{32}}} + {\color{green}{k}} {\color{blue}{b_{32}}} & a_{33} \end{vmatrix} = \begin{vmatrix} a_{11} & {\color{red}{a_{12}}} & a_{13} \\ a_{21} & {\color{red}{a_{22}}} & a_{23} \\ a_{31} & {\color{red}{a_{32}}} & a_{33} \end{vmatrix} + {\color{green}{k}} \begin{vmatrix} a_{11} & {\color{blue}{b_{12}}} & a_{13} \\ a_{21} & {\color{blue}{b_{22}}} & a_{23} \\ a_{31} & {\color{blue}{b_{32}}} & a_{33} \end{vmatrix} \]
- Row and Column Operations:
- Swapping two rows (or columns) of a matrix multiplies the determinant by -1.
- Adding a multiple of one row (or column) to another row (or column) does not change the determinant.
- Multiplying a row (or column) by a scalar multiplies the determinant by that scalar.
- Triangular Matrices: The determinant of a triangular matrix (upper or lower) is the product of its diagonal elements.
\[ \det(\mathbf{A}) = \begin{vmatrix} \color{red}{a_{11}} & \color{blue}{0} & \ldots & \color{blue}{0} \\ \color{orange}{a_{21}} & \color{red}{a_{22}} & \ldots & \color{blue}{0} \\ \vdots & \vdots & \ddots & \vdots \\ \color{orange}{a_{n1}} & \color{orange}{a_{n2}} & \ldots & \color{red}{a_{nn}} \end{vmatrix} = \color{red}{a_{11}} \cdot \color{red}{a_{22}} \cdot \ldots \cdot \color{red}{a_{nn}} \]
- Determinant of the Identity Matrix: The determinant of the identity matrix is 1.
\[ \det(\mathbf{I}) = 1 \]
- Determinant of a Product: The determinant of the product of two matrices is the product of their determinants.
\[ \det(\mathbf{A} \cdot \mathbf{B}) = \det(\mathbf{A}) \cdot \det(\mathbf{B}) \]
- Determinant of the Transpose: The determinant of a matrix is equal to the determinant of its transpose.
\[ \det(\mathbf{A}) = \det(\mathbf{A}^T) \]
- Determinant of an Inverse: If a matrix is invertible, the determinant of its inverse is the reciprocal of the determinant of the matrix.
\[ \det(\mathbf{A}^{-1}) = \frac{1}{\det(\mathbf{A})} \]
- Block Matrices: For block matrices, if \(A\) and \(D\) are square matrices, and \(B\) and \(C\) are zero matrices, then: \[ \text{det}\begin{pmatrix} A & 0 \\ 0 & D \end{pmatrix} = \text{det}(A) \cdot \text{det}(D) \]
These properties can be used to simplify the calculation of determinants, especially for larger matrices.
Example
The initial form of determinant is:
\[ D = \begin{vmatrix} 2 & 3 & 1 & 5 \\ -1 & 1 & -1 & 2 \\ 3 & 0 & 2 & 1 \\ 4 & 1 & 3 & 2 \end{vmatrix} \]
We perform the following column operations:
- Add the second column to the first column.
- Add the second column to the third column.
- Subtract twice the second column from the fourth column.
After these operations,
\[ D = \begin{vmatrix} 5 & 3 & 4 & -1 \\ 0 & {\color{red}1} & 0 & 0 \\ 3 & 0 & 2 & 1 \\ 5 & 1 & 4 & 0 \end{vmatrix} \]
Now, we calculate this determinant by expanding along the second row (as it contains several zeros):
\[ D = (-1)^{2+2}\cdot {\color{red}1} \cdot \det \begin{vmatrix} 5 & 4 & -1 \\ 3 & 2 & 1 \\ 5 & 4 & 0 \end{vmatrix} \]
Now, we will do the same thing as before. Let’s add the second row to the first row, which will give us:
\[ D=\begin{vmatrix} 8 & 6 & 0 \\ 3 & 2 & {\color{blue}1} \\ 5 & 4 & 0 \end{vmatrix} \]
Now we exand along the third column:
\[ D = (-1)^{2+3} \cdot {\color{blue}1}\cdot \det \begin{vmatrix} 8 & 6 \\ 5 & 4 \end{vmatrix} = -1 \cdot (8 \cdot 4 - 6 \cdot 5) = -2 \]
Inverse of a Matrix
The inverse of a matrix \(\mathbf{A}\) is a matrix \(\mathbf{B}\) such that the product of \(\mathbf{A}\) and \(\mathbf{B}\) is the identity matrix \(\mathbf{I}\):
\[ \mathbf{A} \cdot \mathbf{B} = \mathbf{B} \cdot \mathbf{A} = \mathbf{I} \]
we will denote the inverse of a matrix \(\mathbf{A}\) as \(\mathbf{A}^{-1}\).
Not all matrices are invertible. A matrix is invertible if and only if its determinant is non-zero. By definition, the inverse of a matrix is unique and can be calculated only for square matrices.
Expansion Formula for the Inverse
The inverse of a matrix can be calculated using the following formula:
\[ \begin{align*} \mathbf{A}^{-1} &= \frac{1}{\text{det}(\mathbf{A})} \begin{pmatrix} \color{red}{C_{11}} & \color{orange}{C_{12}} & \ldots & \color{blue}{C_{1n}} \\ \color{red}{C_{21}} & \color{orange}{C_{22}} & \ldots & \color{blue}{C_{2n}} \\ \vdots & \vdots & \ddots & \vdots \\ \color{red}{C_{n1}} & \color{orange}{C_{n2}} & \ldots & \color{blue}{C_{nn}} \end{pmatrix}^{T}\\ & \\ &= \frac{1}{\text{det}(\mathbf{A})} \begin{pmatrix} \color{red}{C_{11}} & \color{red}{C_{21}} & \ldots & \color{red}{C_{n1}} \\ \color{orange}{C_{12}} & \color{orange}{C_{22}} & \ldots & \color{orange}{C_{n2}} \\ \vdots & \vdots & \ddots & \vdots \\ \color{blue}{C_{1n}} & \color{blue}{C_{2n}} & \ldots & \color{blue}{C_{nn}} \end{pmatrix} \end{align*} \]
where \(C_{ij}\) is the cofactor of the element \(a_{ij}\) in the matrix \(\mathbf{A}\).
2x2 Matrix
For a \(2 \times 2\) matrix \(\mathbf{A}\):
\[ \mathbf{A} = \begin{pmatrix} a & b \\ c & d \end{pmatrix} \]
The inverse of \(\mathbf{A}\) is:
\[ \mathbf{A}^{-1} = \frac{1}{ad - bc} \begin{pmatrix} d & -b \\ -c & a \end{pmatrix} \]
3x3 Matrix
For a \(3 \times 3\) matrix \(\mathbf{A}\):
\[ \mathbf{A} = \begin{pmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{pmatrix} \]
The cofactors of the elements are:
\[ \begin{align*} C_{11} & = a_{22}a_{33} - a_{23}a_{32} \\ C_{12} & = -(a_{21}a_{33} - a_{23}a_{31}) \\ C_{13} & = a_{21}a_{32} - a_{22}a_{31} \\ C_{21} & = -(a_{12}a_{33} - a_{13}a_{32}) \\ C_{22} & = a_{11}a_{33} - a_{13}a_{31} \\ C_{23} & = -(a_{11}a_{32} - a_{12}a_{31}) \\ C_{31} & = a_{12}a_{23} - a_{13}a_{22} \\ C_{32} & = -(a_{11}a_{23} - a_{13}a_{21}) \\ C_{33} & = a_{11}a_{22} - a_{12}a_{21} \end{align*} \]
so the inverse of \(\mathbf{A}\) is:
\[ \mathbf{A}^{-1} = \frac{1}{\text{det}(\mathbf{A})} \begin{pmatrix} \color{red}{C_{11}} & \color{red}{C_{12}} & \color{red}{C_{13}} \\ \color{blue}{C_{21}} & \color{blue}{C_{22}} & \color{blue}{C_{23}} \\ \color{green}{C_{31}} & \color{green}{C_{32}} & \color{green}{C_{33}} \end{pmatrix}^T = \frac{1}{\text{det}(\mathbf{A})} \begin{pmatrix} \color{red}{C_{11}} & \color{blue}{C_{21}} & \color{green}{C_{31}} \\ \color{red}{C_{12}} & \color{blue}{C_{22}} & \color{green}{C_{32}} \\ \color{red}{C_{13}} & \color{blue}{C_{23}} & \color{green}{C_{33}} \end{pmatrix} \]
To check if our computation is correct, we can multiply the matrix \(\mathbf{A}\) by its inverse \(\mathbf{A}^{-1}\) and verify that the result is the identity matrix \(\mathbf{I}\).
Gauss-Jordan method
Finding the inverse of a matrix can be done using the Gauss-Jordan method. This method involves augmenting the matrix with the identity matrix and performing row operations to transform the original matrix into the identity matrix. The right side of the augmented matrix will then become the inverse matrix.
Example
Let’s find the inverse of the following \(2 \times 2\) matrix \(\mathbf{A}\):
We will perform step-by-step transformations to find the inverse matrix for \(\mathbf{A}\) using the Gauss-Jordan method. We will apply row operations to transform the left side of the augmented matrix into the identity matrix, and the right side will then become the inverse matrix.
If the initial matrix \(\mathbf{A}\) is
\[ \mathbf{A} = \begin{bmatrix} 2 & 3 \\ 4 & 1 \end{bmatrix} \]
we can find the inverse by following the Gauss-Jordan steps with the augmented matrix \(\left[ \mathbf{A} \mid \mathbf{I} \right]\).
Step-by-step solution:
Write the augmented matrix:
\[ \left[ \mathbf{A} \mid \mathbf{I} \right] = \begin{bmatrix} 2 & 3 & \big| & 1 & 0 \\ 4 & 1 & \big| & 0 & 1 \\ \end{bmatrix} \]
Make the first element in the first row a “1” by dividing the first row by 2:
\[ \begin{bmatrix} 1 & \frac{3}{2} & \big| & \frac{1}{2} & 0 \\ 4 & 1 & \big| & 0 & 1 \\ \end{bmatrix} \]
Make the element in the second row, first column a “0” by subtracting 4 times the first row from the second row:
\[ \begin{bmatrix} 1 & \frac{3}{2} & \big| & \frac{1}{2} & 0 \\ 0 & -5 & \big| & -2 & 1 \\ \end{bmatrix} \]
Make the second element in the second row a “1” by dividing the second row by -5:
\[ \begin{bmatrix} 1 & \frac{3}{2} & \big| & \frac{1}{2} & 0 \\ 0 & 1 & \big| & \frac{2}{5} & -\frac{1}{5} \\ \end{bmatrix} \]
Make the element in the first row, second column a “0” by subtracting \(\frac{3}{2}\) times the second row from the first row:
\[ \begin{bmatrix} 1 & 0 & \big| & \frac{1}{10} & \frac{3}{10} \\ 0 & 1 & \big| & \frac{2}{5} & -\frac{1}{5} \\ \end{bmatrix} \]
The right side of the augmented matrix is now the inverse of \(\mathbf{A}\):
\[ \mathbf{A}^{-1} = \begin{bmatrix} \frac{1}{10} & \frac{3}{10} \\ \frac{2}{5} & -\frac{1}{5} \\ \end{bmatrix} \]
Please read about the Computational_efficiency for more information.
Systems of Linear Equations
A system of linear equations is a set of equations that can be written in the form:
\[ \begin{align*} a_{11}x_1 + a_{12}x_2 + \ldots + a_{1n}x_n & = b_1 \\ a_{21}x_1 + a_{22}x_2 + \ldots + a_{2n}x_n & = b_2 \\ \vdots & \\ a_{m1}x_1 + a_{m2}x_2 + \ldots + a_{mn}x_n & = b_m \\ \end{align*} \]
where \(a_{ij}\) are the coefficients of the variables \(x_i\) and \(b_i\) are the constants.
The system can be written in matrix form as: \(b_i\) are the constants, and \(m\) is the number of equations:
\[ \begin{bmatrix} a_{11} & a_{12} & \ldots & a_{1n} \\ a_{21} & a_{22} & \ldots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \ldots & a_{mn} \\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \\ \end{bmatrix} = \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_m \\ \end{bmatrix} \]
Example
Consider the following system of linear equations:
\[ \begin{align*} 2x + 2y & = 6 \\ 4x - y & = 7 \end{align*} \]
This system can be written in matrix form as:
\[ \begin{bmatrix} 2 & 2 \\ 4 & -1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 6 \\ 4 \end{bmatrix} \]
One can check that the solution to this system is \(x = 2\) and \(y = 1\).
Cramer’s Formula
Cramer’s rule is a method for solving systems of linear equations using determinants. For a system of \(n\) linear equations with \(n\) variables, the solution can be expressed using determinants.
Conditions:
- If \(\det(A) \neq 0\): Unique solution exists.
- If \(\det(A) = 0\) and \(\det(A_i) \neq 0\) for at least one \(i\): No solution.
- If \(\det(A) = 0\) and \(\det(A_i) = 0\) for all \(i\): Infinite solutions.
Given a system of linear equations:
\[ \begin{align*} a_{11}x_1 + a_{12}x_2 + \ldots + a_{1n}x_n & = b_1 \\ a_{21}x_1 + a_{22}x_2 + \ldots + a_{2n}x_n & = b_2 \\ \vdots & \\ a_{n1}x_1 + a_{n2}x_2 + \ldots + a_{nn}x_n & = b_n \\ \end{align*} \]
The solution for the variable \(x_i\) is given by:
\[ x_i = \frac{\text{det}(\mathbf{A}_i)}{\text{det}(\mathbf{A})} \]
where \(\mathbf{A}\) is the coefficient matrix of the system, and \(\mathbf{A}_i\) is the matrix obtained by replacing the \(i\)-th column of \(\mathbf{A}\) with the column vector \(\mathbf{b}\).
Example - 2x2
\[ \begin{align*} 2x + 2y & = 6 \\ 4x - y & = 7 \end{align*} \]
We have to comute the determinants of the matrices:
\[ \begin{align*} \text{det}(\mathbf{A}) &= \begin{vmatrix} 2 & 2 \\ 4 & -1 \end{vmatrix} = 2 \cdot (-1) - 2 \cdot 4 = -2 - 8 = -10 \\ \text{det}(\mathbf{A}_1) &= \begin{vmatrix} 6 & 2 \\ 7 & -1 \end{vmatrix} = 6 \cdot (-1) - 2 \cdot 7 = -6 - 14 = -20 \\ \text{det}(\mathbf{A}_2) &= \begin{vmatrix} 2 & 6 \\ 4 & 7 \end{vmatrix} = 2 \cdot 7 - 6 \cdot 4 = 14 - 24 = -10 \end{align*} \]
so the solution is:
\[ x = \frac{\text{det}(\mathbf{A}_1)}{\text{det}(\mathbf{A})} = \frac{-20}{-10} = 2 \]
\[ y = \frac{\text{det}(\mathbf{A}_2)}{\text{det}(\mathbf{A})} = \frac{-10}{-10} = 1 \]
Example - 3x3
Consider the following system of linear equations:
\[ \begin{align*} x + 2y + 3z & = 9 \\ 2x - y + z & = 8 \\ 3x + y - 2z & = 3 \end{align*} \]
We first form the coefficient matrix \(\mathbf{A}\), as well as the matrices \(\mathbf{A}_1\), \(\mathbf{A}_2\), and \(\mathbf{A}_3\) obtained by replacing the respective columns with the constants on the right-hand side:
\[ \begin{align*} \det(\mathbf{A}) &= \begin{vmatrix} 1 & 2 & 3 \\ 2 & -1 & 1 \\ 3 & 1 & -2 \end{vmatrix}=30\\ \det(\mathbf{A}_1) &= \begin{vmatrix} 9 & 2 & 3 \\ 8 & -1 & 1 \\ 3 & 1 & -2 \end{vmatrix}=80\\ \det(\mathbf{A}_2) &= \begin{vmatrix} 1 & 9 & 3 \\ 2 & 8 & 1 \\ 3 & 3 & -2 \end{vmatrix}=-10\\ \det(\mathbf{A}_3) &= \begin{vmatrix} 1 & 2 & 9 \\ 2 & -1 & 8 \\ 3 & 1 & 3 \end{vmatrix}=70 \end{align*} \]
so the solution is:
\[ \begin{align*} x &= \frac{\det(\mathbf{A}_1)}{\det(\mathbf{A})} = \frac{80}{30} = \frac{8}{3} \\ y &= \frac{\det(\mathbf{A}_2)}{\det(\mathbf{A})} = \frac{-10}{30} = -\frac{1}{3} \\ z &= \frac{\det(\mathbf{A}_3)}{\det(\mathbf{A})} = \frac{70}{30} = \frac{7}{3} \end{align*} \]
Gaussian Elimination
Classical Method
Let’s consider the following system of linear equations:
\[ \begin{align*} x + y + z &= 6 \\ 2x - y + z &= 1 \\ -x + 2y + 3z &= 11 \end{align*} \]
We want to solve this system using Gaussian elimination. The first step is to write the augmented matrix:
Step 1: Add 1st row to 3rd row
\[ \begin{align*} x + y + z &= 6 \\ 2x - y + z &= 1 \\ 0x + 3y + 4z &= 17 \end{align*} \]
Step 2: Subtract 2 times the 1st row from the 2nd row
\[ \begin{align*} x + y + z &= 6 \\ 0x - 3y - z &= -11 \\ 0x + 3y + 4z &= 17 \end{align*} \]
Step 3: Add the 2nd row to the 3rd row
\[ \begin{align*} x + y + z &= 6 \\ 0x - 3y - z &= -11 \\ 0x + 0y + 3z &= 6 \end{align*} \]
Now we can solve the system of equations:
\[ 3z = 6 \Rightarrow z = 2 \]
Substitute \(z = 2\) into the second equation:
\[ -3y - 2 = -11 \Rightarrow y = 3 \]
Substitute \(y = 3\) and \(z = 2\) into the first equation:
\[ x + 3 + 2 = 6 \Rightarrow x = 1 \]
Matrix Form of a System of Equations
The system of linear equations can be written in matrix form as:
We want to solve this system using Gaussian elimination. The first step is to write the augmented matrix:
\[ \begin{pmatrix} 1 & 1 & 1 & | & 6 \\ 2 & -1 & 1 & | & 1 \\ -1 & 2 & 3 & | & 11 \end{pmatrix} \]
Step 1: Add 1st row to 3rd row
\[ \begin{pmatrix} 1 & 1 & 1 & | & 6 \\ 2 & -1 & 1 & | & 1 \\ 0 & 3 & 4 & | & 17 \end{pmatrix} \]
Step 2: Subtract 2 times the 1st row from the 2nd row
\[ \begin{pmatrix} 1 & 1 & 1 & | & 6 \\ 0 & -3 & -1 & | & -11 \\ 0 & 3 & 4 & | & 17 \end{pmatrix} \]
Step 3: Add the 2nd row to the 3rd row
\[ \begin{pmatrix} 1 & 1 & 1 & | & 6 \\ 0 & -3 & -1 & | & -11 \\ 0 & 0 & 3 & | & 6 \end{pmatrix} \]
Now we can solve the system of equations:
\[ 3z = 6 \Rightarrow z = 2 \]
Substitute \(z = 2\) into the second equation:
\[ -3y - 2 = -11 \Rightarrow y = 3 \]
Substitute \(y = 3\) and \(z = 2\) into the first equation:
\[ x + 3 + 2 = 6 \Rightarrow x = 1 \]
Inverse Matrix Method
The system of linear equations can be written in matrix form as:
\[ \begin{bmatrix} a_{11} & a_{12} & \ldots & a_{1n} \\ a_{21} & a_{22} & \ldots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \ldots & a_{mn} \\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \\ \end{bmatrix} = \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_m \\ \end{bmatrix} \]
in short \[ \mathbf{A} \mathbf{x} = \mathbf{b} \]
where \(\mathbf{A}\) is the coefficient matrix, \(\mathbf{x}\) is the vector of variables, and \(\mathbf{b}\) is the vector of constants.
By multiplying both sides of the equation by the inverse of the coefficient matrix \(\mathbf{A}^{-1}\), we can solve for the vector of variables \(\mathbf{x}\):
\[ \mathbf{A}^{-1}\mathbf{A}\mathbf{x} = \mathbf{A}^{-1} \mathbf{b} \]
so
\[ \mathbf{x} = \mathbf{A}^{-1} \mathbf{b} \]
Example
Consider the following system of linear equations:
\[ \begin{align*} x + y + z &= 6 \\ 2x - y + z &= 1 \\ -x + 2y + 3z &= 11 \end{align*} \]
this system can be written in matrix form as:
\[ \begin{bmatrix} 1 & 1 & 1 \\ 2 & -1 & 1 \\ -1 & 2 & 3 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} 6 \\ 1 \\ 11 \end{bmatrix} \]
Inverse of the matrix \(\mathbf{A}\) is:
\[ \mathbf{A}^{-1} = \begin{bmatrix} \frac{5}{9} & \frac{1}{9} & -\frac{2}{9} \\ \frac{7}{9} & -\frac{4}{9} & -\frac{1}{9} \\ -\frac{1}{3} & \frac{1}{3} & \frac{1}{3} \end{bmatrix} \]
by multiplying the inverse matrix by the vector of constants, we can find the solution:
\[ \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} \frac{5}{9} & \frac{1}{9} & -\frac{2}{9} \\ \frac{7}{9} & -\frac{4}{9} & -\frac{1}{9} \\ -\frac{1}{3} & \frac{1}{3} & \frac{1}{3} \end{bmatrix} \begin{bmatrix} 6 \\ 1 \\ 11 \end{bmatrix} = \begin{bmatrix} \frac{1}{9}(5*6 + 1*1 - 2*11) \\ \frac{1}{9}(7*6 - 4*1 - 1*11) \\ \frac{1}{3}(-1*6 + 1*1 + 1*11) \end{bmatrix} = \begin{bmatrix} 1 \\ 3 \\ 2 \end{bmatrix} \]