Mathematics > Linear Algebra > Singular Value Decomposition

Singular Value Decomposition (SVD) is a fundamental technique in linear algebra, pivotal in various applications across scientific computing, data analysis, and machine learning. SVD provides a robust method for decomposing a real or complex matrix into a product of three simpler matrices, revealing intrinsic geometric and algebraic properties of the original matrix.

Given any \( m \times n \) matrix \( \mathbf{A} \), the Singular Value Decomposition is expressed as:
\[ \mathbf{A} = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^T \]
where:

• \( \mathbf{U} \) is an \( m \times m \) orthogonal matrix. The columns of \( \mathbf{U} \) are known as the left singular vectors of \( \mathbf{A} \).
• \( \mathbf{\Sigma} \) is an \( m \times n \) diagonal matrix with non-negative real numbers on the diagonal. These numbers are called the singular values of \( \mathbf{A} \). The singular values are typically arranged in descending order.
• \( \mathbf{V}^T \) (or \( \mathbf{V}^\dagger \), in the complex case) is the transpose (or conjugate transpose) of an \( n \times n \) orthogonal matrix \( \mathbf{V} \). The columns of \( \mathbf{V} \) are referred to as the right singular vectors of \( \mathbf{A} \).

Properties and Interpretation:

1. Orthogonality and Diagonalization:
   • The matrices \( \mathbf{U} \) and \( \mathbf{V} \) are orthogonal, meaning \( \mathbf{U}^T \mathbf{U} = \mathbf{I} \) and \( \mathbf{V}^T \mathbf{V} = \mathbf{I} \), where \( \mathbf{I} \) is the identity matrix. This assures that the singular vectors are orthonormal.
2. Geometric Interpretation:
   • SVD can be seen as decomposing \( \mathbf{A} \) into three transformations: \( \mathbf{V}^T \) rotates the original space, \( \mathbf{\Sigma} \) scales it along the axes, and \( \mathbf{U} \) applies another rotation to align the space with the new coordinates.
3. Rank and Pseudoinverse:
   • The number of non-zero singular values of \( \mathbf{A} \) is equal to the rank of \( \mathbf{A} \).
   • SVD provides a means to compute the pseudoinverse of a matrix, especially useful for solving least squares problems.

Applications:

• Principal Component Analysis (PCA):
  SVD plays a crucial role in PCA, a statistical technique used to emphasize variation and identify the principal directions in a dataset. By applying SVD to the covariance matrix of the data, one can determine the principal components.

• Data Compression:
  By truncating the smaller singular values, one can approximate the original matrix with reduced dimensions, thus compressing the data while retaining its most significant features.

• Signal Processing:
  In signal processing, SVD is used for noise reduction and for constructing low-rank approximations.

• Regularization in Machine Learning:
  SVD helps in addressing issues of multicollinearity and ill-posedness in regression models.

The elegance and versatility of Singular Value Decomposition make it a powerful tool in theoretical and applied contexts, offering insight into the structure of matrices and facilitating various computational techniques.