Foundations of Machine Learning - Module 1 (LINEAR ALGEBRA )

Foundations of
Machine Learning
MODULE 1
ASHMILA K P
Prepared By,

Systems of Linear Equations
● A system of linear equations is a collection of two or more linear
equations involving the same set of variables.
● The goal is to find the values of the variables that satisfy all the
equations simultaneously.
● General Form

● A homogeneous system of linear equations is a system where all
constant terms are zero.
● In general, it takes the form:
● If atleast one constant term is not zero, Then it is called Non
homogeneous system of linear equations

● a11, a12, ………. are called Coefficients of the system
● x1, x2, ……….. are called Solutions of the system
● If the system of equations has one or more solutions, then it is
called Consistent
● If the system of equations has no solutions, then it is called In
Consistent

MATRICES
● A matrix is a rectangular array of numbers, symbols, or
expressions arranged in rows and columns.
● Each element in a matrix is identified by its position using two
indices: row number and column number.

1. Row and Column Matrix
➔ A column matrix is a matrix that has only one column and any
number of rows. Its order is m×1 times , where m is the number of
rows.
➔ A row matrix is a matrix that has only one row and any number of
columns. Its order is 1×n times, where n is the number of columns.
Diﬀerent Types of Matrices

2. Square Matrix
➔ A square matrix is a matrix that has an equal number of rows and
columns.
➔ Its order is n×n times, where n is the number of rows (or columns).

3. Diagonal Matrix
➔ A diagonal matrix is a square matrix in which all the elements
outside the main diagonal are zero.
➔ The elements on the main diagonal can be any value, including zero.

4. Upper triangular and Lower triangular Matrix
➔ An upper triangular matrix is a square matrix in which all the
elements below the main diagonal are zero. The elements on and
above the main diagonal can be any value.
➔ A lower triangular matrix is a square matrix in which all the
elements above the main diagonal are zero. The elements on and
below the main diagonal can be any value.

UPPER TRIANGULAR MATRIX LOWER TRIANGULAR MATRIX

5. Scalar Matrix
➔ A scalar matrix is a special type of diagonal matrix where all the
diagonal elements are equal.

6. Identity Matrix
➔ An identity matrix is a special square matrix in which all the
elements of the main diagonal are 1, and all other elements are 0.
➔ It is denoted by I

7. Zero Matrix or Null Matrix
➔ A zero matrix (also called a null matrix) is a matrix in which all the
elements are zero.

OPERATIONS ON MATRICES
1. Determinant
The determinant is a scalar value that can be computed from the
elements of a square matrix.
2x2 Matrix
3x3 Matrix

2. Transpose
The Transpose of a matrix is a new matrix obtained by swapping
the rows and columns of the original matrix.

3. Minor
➔ The minor of an element in a matrix is the determinant of the
submatrix that remains after removing the row and column in which
the element is located.
➔ The minor of an element is denoted as Mij, where i is the row number
and j is the column number of the element.
➔ To calculate the minor of an element aij, you remove the i-th row and
the j-th column from the matrix, and then calculate the determinant of
the remaining smaller matrix.

4. Cofactor
➔ The cofactor of an element aij in a matrix is the signed minor of that
element.

5. Adjoint
➔ The adjoint (or adjugate) of a matrix is the transpose of its cofactor
matrix.
➔ In other words, if you have a matrix A, the adjoint of A, denoted as
Adj(A)

5. Inverse
➔ The inverse of a square matrix A is another matrix, denoted A−1

STEPS,
1. Determine the minors of all A elements.
2. Next, compute the cofactors of all elements and build the cofactor
matrix by substituting the elements of A with their respective
cofactors.
3. Take the transpose of A’s cofactor matrix to find its adjoint (written
as adj A).
4. Multiply adj A by the reciprocal of the determinant of A.

RANK OF MATRIX
● The rank of a matrix is equal to the number of linearly independent
rows (or columns) in it.
● STEPS,

ROW OPERATIONS
Row swapping: Ri↔Rj (swap rows i and j)
Row scaling: Ri→kRi (multiply row i by a constant k)
Row replacement: Ri→Ri+kRj (replace row i with the sum
of rows i and k times row j)

MATRIX REPRESENTATION OF
SYSTEM OF LINEAR EQUATIONS
This system can be written as:
A X=B
Where,
● A is the matrix of coefficients (of size m×n),
● X is the column matrix of variables (of size n×1),
● B is the column matrix of constants (of size m×1).

AUGMENTED MATRIX
➔ An augmented matrix is a matrix that represents a system
of linear equations by combining the coefficient matrix and
the constants into a single matrix.
➔ It is formed by appending the constants (from the
right-hand side of the equations) as an extra column to the
coefficient matrix.

CONSISTENT SYSTEM
➔ The system is consistent if the rank of the coefficient matrix
(A) is equal to the rank of the augmented matrix (A|B).
➔ Mathematically: rank(A)=rank(A∣B)
➔ The system is inconsistent if the rank of the coefficient matrix (A)
is less than the rank of the augmented matrix (A|B).
➔ Mathematically: rank(A)≠rank(A∣B)
INCONSISTENT SYSTEM

QUESTION
Check whether the following system of equations is consistent or
inconsistent.
2x-2z = 6
y+z = 1
2x+y-z = 7
3y+3z =0

TUTORIAL QUESTIONS
1) Show that the equations
x+2y+z = 0
3x+2y+z = 2
2x-y+2z = 5
5x+6y+3z = 2
X+3y-z = -3
Are consistent and solve the same.

2) Show that the equations
y-3z = -1
x+z = 1
3x+y = 2
x+y-2z = 0
Are consistent and solve the same.

Solving Systems of Linear Equations
Gauss Elimination Method
Steps,
1. Write the augmented matrix for the system of linear equations.
2. Forward Elimination: Use row operations to transform the matrix
into an upper triangular form (where all elements below the main
diagonal are zeros).
3. Back Substitution: Once the matrix is in upper triangular form, use
back-substitution to solve for the variables.

QUESTION
Calculate all solutions to the system of linear equations.
x+2y+z = 3
2x+3y+2z = 5
3x-5y+5z = 2
3x+9y-z = 4

TUTORIAL QUESTIONS
1) Calculate all solutions to the system of linear
equations.
-x+2y+3z = -2
2x-5y+z = 2
3x-8y+5z = 2
5x-12y-z = 6

equations.
x+2y+z = 0
3x+2y+z = 2
2x-y+2z = 5
5x+6y+3z = 2
x+3y-z = -3

equations.
2x-2z = 6
y+z = 1
2x+y-z = 7
3y+3z = 0

Vector Space
Vector : A vector is an ordered collection of numbers, called
Components, that describes a point in space or a direction relative to
an origin.
Vector Space:
● It is a Set of Vectors.
● A vector space is just a collection of vectors where you can:
1) Add vectors together, and (Adding)
2) Multiply vectors by numbers (scalar Multiplication)

Addition Axioms (Properties)
1. Closure : If u and v are vectors in the space, then u+v is also
in the space.
2. Commutativity : u+v = v+u.
3. Associativity : (u+v)+w = u+(v+w).
4. Identity Element : There is a vector 0 such that v+0=v for any
vector v.
5. Inverse Element : For every vector v, there is a vector −v such
that v+(−v)=0.

Scalar Multiplication Axioms
(Properties)
1. Closure : If v is a vector and c is a scalar, then cv is also in the
space.
2. Distributive property of scalar multiplication over vector
addition: c(u+v)=cu+cv.
3. Distributive property of scalar multiplication over vector
addition: c(u+v)=cu+cv.
4. Associativity : a(bv)=(ab)v.
5. Identity Element : 1v=v for any vector v.

Subspace
● A subspace of a vector space is a subset of that vector space that is itself a vector space,
meaning it satisfies the properties of a vector space. A subspace must adhere to the following
conditions:
● Let V be a vector space over a field (such as, the real numbers), and let W be a subset of V.
For W to be a subspace of V, the following conditions must hold:
1. The zero vector is in W: The zero vector 0 of the vector space V must be in W. This is
because the zero vector is the additive identity in the vector space, and every subspace must
contain this element.
2. Closed under vector addition: If u and v are vectors in W, then u+v must also be in W. This
ensures that the sum of any two vectors in the subspace is also contained within the subspace.
3. Closed under scalar multiplication: If v is a vector in W and c is a scalar, then cv must also
be in W. This means that multiplying any vector in the subspace by a scalar should result in a
vector that is still in the subspace.

Linear Independence
A Set of Vectors {v1,v2,........,vn} is linearly Independent if
the only solution to the equation,
c1v1+c2v2+c3v3+........+cnvn = 0
is,
c1=c2=c3=........=cn = 0
Otherwise the vectors are linearly dependent.

Linear Independence - STEPS
1. Form a matrix with the vectors as columns.
2. Apply row operations to reduce the matrix to row echelon
form.
3. Check if there is a row of zeros.
4. If there is a row of zeros, the vectors are linearly dependent;
otherwise, they are linearly independent.

❖ If the row echelon form has a pivot element in every column
—> Linearly Independent
❖ If there is a zero row or missing pivot, then the vectors are
—> Linearly Dependent

QUESTION
1) Prove whether the following sets of vectors are
linearly independent or dependent:
1) u = [1,2,3] v = [4,5,6] w = [7,8,9]
2) u = [1,0,0] v = [0,1,0] w = [0,0,1]
3)v1 = [1,2,1] v2 = [2,3,4] v3 = [3,6,5]

TUTORIAL QUESTION
1) Prove whether the following sets of vectors are
linearly independent or dependent:
1) u = [1,1,1] v = [1,2,3] w = [1,4,9]
2) u = [1,2,3] v = [2,4,6] w = [3,6,9]
3)v1 = [1,1,0] v2 = [2,3,1] v3 = [3,4,1]

Basis
● A basis of a vector space is a set of linearly independent vectors that span the
entire space.
● In other words, every vector in the space can be written as a linear combination of
the vectors in the basis.
A basis for a vector space is a set of vectors that:
1. Spans the vector space: The vectors in the basis can be combined (via scalar
multiplication and addition) to form any vector in the space.
2. Are linearly independent: No vector in the basis can be written as a combination
of the others.

BASIS - STEPS
1. Form the matrix
2. Reduce the matrix to Row Echelon Form
3. Check for Linear Independence
4. Identify Pivot Columns
5. Form Column Basis : Formed by the columns from the original
matrix that correspond to the pivot columns in the row echelon
form.
6. Identify Non-zero Rows
7. Form Row Basis : Formed by the non-zero rows in the Row
Echelon Form.

QUESTION
1) Find the Column and Row Basis of the given Matrix,

TUTORIAL QUESTIONS

TUTORIAL QUESTIONS
i. Find the row echelon form of matrix B.
ii. Identify the pivot columns and find the column basis of matrix B.
iii. Find the row basis of matrix B using the row echelon form.

Standard Basis vectors
● Standard basis vectors are the basic vectors in a space that
point along each axis and have a magnitude of 1.
● These vectors are called "standard" because they are the
simplest vectors in each direction, and any other vector in that
space can be written as a combination of these basis vectors.

● In a 2D space, the standard basis vectors are:
e1=(1,0) , e2=(0,1)
● In a 3D space, the standard basis vectors are:
e1=(1,0,0)
e2=(0,1,0)
e3=(0,0,1)

Linear Transformation
(Linear Mapping)
● A linear transformation in linear algebra is a function that maps vectors from one
vector space to another, while preserving the operations of vector addition and
scalar multiplication.
● This means that a linear transformation T satisfies the following two properties for
all vectors u,v and scalars c:
a. Additivity:
T(u+v)=T(u)+T(v)
b. Homogeneity (or Scalar multiplication):
T(cu)=cT(u)
● The notation T:V→W represents a linear transformation T from a vector space V
to a vector space W.

Transformation Matrix
● A transformation matrix is a matrix that represents a linear transformation
from one vector space to another.
Steps to Find the Transformation Matrix Using Standard Basis Vectors:
1. Define the linear transformation T from V to W.
2. Apply the transformation to each standard basis vector of V. That is,
calculate T(e1),T(e2),…
3. Form the transformation matrix: The columns of the matrix will be the
results of applying the transformation T to the standard basis vectors,
expressed in terms of the standard basis of W.

Steps to Find the Transformation Matrix: (By considering new basis
vectors)
1. Choose a basis for V: Let {v1,v2,…,vn} be a basis for the domain vector space V.
2. Apply T to each basis vector: Compute T(vi) for each basis vector vi in V.
3. Express T(vi) in terms of the basis of W: Each T(vi) is a vector in W, and you
express it as a linear combination of the basis vectors of W. Suppose the basis of W
is {w1,w2,…,wm}, then express:
T(vi)=ai1w1+ai2w2+⋯+aimwm
where aij are the coefficients.
4. Form the matrix: The matrix A is composed of the coefficients aij as its columns

Kernel
● The kernel (also called the null space) of a linear transformation T:V→W
is the set of all vectors in V that are mapped to the zero vector in W.
Mathematically, it is defined as:
Ker(T)={v∈V ∣ T(v)=0}

STEPS
Steps to compute the kernel: (Transfromation matrix)
1. Set up the equation Av=0
Write the matrix equation as a system of linear equations.
2. Row reduce the augmented matrix [A∣0]] to row echelon form (REF)
3. Solve the resulting system:
○ Use back-substitution to express the solution in terms of free variables.
○ The solution will be a set of vectors that spans the kernel.

● The range (also called the image) of a linear transformation T:V→W is the
set of all possible outputs (vectors in W) that T can produce. It is defined
as:
Range(T)={T(v) ∣ v∈V}
● The range tells us how much of the output space W is covered by the
transformation.
● The dimension of the range is called the rank of the transformation.
Range

Steps to compute the range:
1. Find the pivot columns of the matrix A by row reducing it to row echelon
form (REF).
2. Identify the linearly independent columns:
○ The columns of A corresponding to the pivot columns in REF form a
basis for the range.
3. The range is the span of these linearly independent columns:
○ If the pivot columns are c1,c2,…,ck, then the range is:
Range(T)=span{c1,c2,…,ck}.
STEPS

QUESTION
Define the kernel and range of a linear transformation. For the linear
transformation T: R^3 → R^3 defined by T(x,y,z)=(x+2y+3z , y+4z),
(a)Calculate the kernel of T and express it in parametric form.
(b) Determine the range of T .

TUTORIAL QUESTIONS
1) Let T:R^3 →R^3 be a linear transformation represented by the matrix:
(a) Find the kernel of T and express it in parametric form.

TUTORIAL QUESTIONS
1) Let T:R^3 →R^2 be a linear transformation defined by,
T(V) = { x+2y , 3x+4y }
(a) Find the kernel of T and express it in parametric form.
2) Find the Kernel and range of the matrix given below,

Rank and Nullity
Nullity:
● The nullity of a linear transformation T:V→W is the dimension of
the kernel (or null space) of T.
● The kernel consists of all vectors in V that are mapped to the zero
vector in W.
● The nullity tells us how many independent vectors in V get mapped
to the zero vector in W.
● Nullity(T) = dim(Ker(T))

Rank :
● The rank of a linear transformation T:V→W (where V and W are vector
spaces) is the dimension of the image (or range) of the transformation.
● The image is the set of all vectors in W that are mapped to by vectors from
V.
● The rank tells us how many independent vectors in W are the result of
applying the transformation to vectors in V.
● Rank(T) = dim(Range(T))

Rank and Nullity Theorem
The Rank-Nullity Theorem is a fundamental result in linear algebra that relates the rank and
nullity of a linear transformation.
It states:
Rank(T)+Nullity(T)=dim(V)
Where:
● Rank(T) is the dimension of the image (or column space) of the linear transformation T.
● Nullity(T) is the dimension of the kernel (or null space) of the linear transformation T.
● dim(V) is the dimension of the domain V, the vector space from which the
transformation T takes its input.

QUESTION
1. Consider the Linear transformation T: R^3 →
R^2.Given the linear transformation T defined by,
T(X,Y,Z) = ( X+Y , Y+Z)
Find Rank and nullity?

2. Consider the Linear transformation T: R^3 →
R^2.Given the linear transformation T defined by,
T(X1, X2, X3) = ( X1+X2 , 2X1+3X3)
Find Rank and nullity?

TUTORIAL QUESTIONS
1. Let T: R^3 → R^2 be a linear mapping represented by
T(x,y,z)=(x+y , 2y+z).
1. Determine the nullity of T.
2. Calculate the rank of T.
2. Let T: R^4 → R^3 be a linear mapping represented by
Calculate the rank and nullity of T.

Foundations of Machine Learning - Module 1 (LINEAR ALGEBRA )

More Related Content

Similar to Foundations of Machine Learning - Module 1 (LINEAR ALGEBRA )

Recently uploaded

Foundations of Machine Learning - Module 1 (LINEAR ALGEBRA )