Linear Algebra: Inner Products MathTheBeautiful
Chapter 1
Motivation
Definition of inner product
examples
We need to define somhow the concept of length. Very practical when talking about aproximation
Dot product = inner product for geometric vectors
lengh and angles are given
First exapmle of use in projections
Pb(a)=⟨a,b⟩⟨b,b⟩b
\(lengh(a)=\sqrt{\langle a,a\rangle }\)
\( \langle a, b\rangle=lengh(a)lengh(b)\cos (a,b) \)
For ortonormal basis \( {e_i} \) we have
\( v=v^ie_i \rightarrow v^i=\langle v, e_i\rangle\)
\( a \perp b \leftrightarrow \langle a,b\rangle =0\)
Inner products comes first,
lengh and angles comes from inner products
Definition of inner products ‼
wikipedia
- Linearity
\( \langle ax,y\rangle=a\langle x,y\rangle \) - Conjugate symmetry (symmetry in real case)
\( \langle x,y \rangle=\langle y,x\rangle \)
3.Positive-definiteness
\( \langle x,x\rangle \geqslant 0 \)
\( \langle x,x\rangle =0 \leftrightarrow x=0 \)
Geometric vectors
\( \langle a, b\rangle=lengh(a)lengh(b)\cos (a,b) \)
Polynomials
\( \langle p,q\rangle=\int_{-1}^1 p(x)q(x)dx \)
you can thin of this def. as infinite sum in
standart inner product
standart inner product in\( \mathbb{R}^n \)
\( \langle x,y\rangle =x_1y_1+..+x_ny_n\)
Gram-Schmidt Orthogonalization
\( u_i=v_i-\sum\limits_k \frac{\langle v_i, u_k\rangle}{\langle u_k ,u_k \rangle}u_k\)
Legendre Polynomials polynomials that are
produced by ortogonalization procedure of the standart basie
12.Gram-Schmidt Orthogonalization and Legendre Polynomials
13,Decomposition with Respect to Legendre Polynomials
14.Why {1,x,x²} Is a Terrible Basis
From definition after normalization
\( \int_{-1}^1 P_i(x)P_j(x)dx=\sigma_i^j\)
Rodrigues' formula:
\(P_n (x)=\frac{1}{2^n n!}\frac{d}{dx}[(x^2-1)^n]\)
ortogonality of the basis implay better aproximations
small error in input implay small error in outputs
in order to measure error you need lengh that tells you
your inner product
QR Decomposition=
the matrix representation
of Gram-Schmit ortogonaization ⚠
QR decomposition of a matrix A into a product A = QR of an orthogonal matrix Q and an upper triangular matrix R. QR decomposition is often used to solve the linear least squares problem, and is the basis for a particular eigenvalue algorithm, the QR algorithm.
Gaussian Quadrature ⚠
we need to figure out how to choose wights smart
and how to choose smart \(x_i\)
Most numerical integration procedure looks like
\( \int_{-1}^1 f dz= w_1 f(x_1)+w_2f(x_2)+...+w_nf(x_n)\)
this problem gives motivation why we are considering
an "integral" inner product, and wy Legendre polynomials
are mode to be ortogonal with respect to this inner product
Step1. we devide 2n-1 polynomial by Lagrange polynomial
\( \int_{-1}^1 p(x)dx=\int_{-1}^1q(x)L_n(x)dx+\int_{-1}^1r(x)dx \)
Step2. We choose \(x_i\) as zeros of \(L_n \) to make first integral exacly, we choose weights \(w_i\)to make second integral exaclly
First integral is equal zero becouse \(L_n \) is perpendicular
to polynomials of degre n-1 or less (and \( deg q \leqslant n-1 \) )
Chapter 2
Matrix Representation of Inner product
Transport apear when you use inner
product in matrix form ⚠
Inner product implays that matrix has to be symetric
Linearity of inner product implays marix representation
Positive defnite of inner product implay positive def of a matrix
thats just synonim
If matrix is positive than the diagonal coefficient have to be positive
For a 2 by 2 matrix you can consider a quadratic function
of 1 variable (can always skale such that y=1) and can came
up with general condition
\(a_{11}>0 \) , \(\det A>0 \)
Sylvester's criterion states that a Hermitian matrix M is positive-definite if and only if all of the leading principal minors must be positive.
LDU =LDLᵀ if and only
if matrix is symetric
For a matrix to be positive definite, all the pivots
of the matrix should be positive ⚠
Pivots are the first non-zero element in each row of a matrix that is in Row-Echelon form. Row-Echelon form of a matrix is the final resultant matrix of Gaussian Elimination technique.
Eigenvectors of Symmetric Matrices Are Orthogonal
If all the Eigen values of the symmetric matrix are positive, then it is a positive definite matrix.
Minimization
\(f''(x)>0 \)
is equivalent in multivariable case is
matrix \(H\) (hessian) is positive define
that comes from "\(f''(t)>0\)"
for all lines that goes from x
Critical point occures when \(f'(x)=0\)
is equivalent in multivariable case is
\(\nabla f(x)=0\)
that comes from "\(f'(t)=0\)"
for all lines that goes from x
Least squares formula for
overdetermined system \(Ax=B\)
\(x=(A^{T}A)^{-1}A^{t}B\)
Simp. to \(x=A^{-1}B\) if \(A^{-1}\) exists
comes from minimalizing the length \(r^{T}r\),
where \(r=B-Ax\) is an "error" vector
You can solve aproximatly system
\(Ax=B\)
for positive def matrix A, by minimalieze function
\(f(x)=1/2x^{T}Ax-x^{T}B\)
Chapter 3
Decomposition
positive deff. can be define for non symetric/hermitian matrix
-you can view a Fourier Series problem
as a Least Squers problem with ortogonal basie
\( \sin nx, cosnx, n=1,..,k\)
that span some subspeces of function's space
-if yo accept infinite sums you can also see Fourier Series problem as a decomposition problem with respect to \( \sin nx, cosnx, n=1,..,+\infty\) basie
-you can solve the decomposition problem by considering \( \langle v, e_i \rangle \)
-ortonormal basie gives you the easiest calculations, ortogonal base is easy as well, in normal base case you need to consider the whole system \(Ax=B\)
-Metric \( [\langle e_i, e_j\rangle]_{i,j}=A^TA
, [\langle v, e_j\rangle]_{i,j}=A^TB\)
-this gives the least squares formula \(x=(A^TA)^{-1}A^TB\)
When you can not solve \(Ax=B\)
it is succesful to multiplay both sides of system by \(A^T\)
(you can think of that operation as a prodjection of \( B\) on the
space span by columns of \( A\) and solve \(A^TAx=A^TB\)
In "Cramer" case it just happen that
\(B\) is already in column space of \(A\)