Fdtxjr

My question

Would the $theta$ be still less than 90 degrees in v^T * M^k v = ||v|| * ||M^k v|| * cos $theta$, if the matrix M is positive definite?

Background Information

• Let's suppose that v (original vector: v) is a non-zero vector and M is a positive definite matrix. I multiply M several K times to a vector v (mapped vector : M^k v ) and then inner product between those two vectors. The mathematical equation is as follows :

         v^T * M^k v = ||v|| * ||M^k v|| * cos $theta$

• 
  v^T * M v >0 : This is a sure thing because M is a positive definite matrix. As such, the angle $theta$ between the v (original vector) and the Mv (the mapped vector) is less than 90 degree, since the inner product should be positive.


• I am curious to know if the $theta$ in the equation below is also between 0 degree and 90 degree.

         v^T * M^k v = ||v|| * ||M^k v|| * cos $theta$

• I got motivated to ask this question from the Towards Data Science article 'What is a Positive Definite Matrix?'. The quote is as follows:

Wouldn’t it be nice in an abstract sense… if you could multiply some matrices multiple times and they won’t change the sign of the vectors? If you multiply positive numbers to other positive numbers, it doesn’t change its sign. I think it’s a neat property for a matrix to have.

If $M$ is a positive definite matrix, then so is $M^k$, for all integers $k$ (note that negative powers of $M$ also exist since $|M| ne 0$).

This is easy to see. A matrix $A$ is positive definite if and only if all its eigenvalues are positive [additionally, $A$ may also be required to be symmetric, in the more common definition]. If $lambda_1, ldots, lambda_n$ are all the eigenvalues of a matrix $M$, then eigenvalues of $M^k$ are exactly $lambda_1^k, ldots, lambda_n^k$ (this also holds for negative $k$ if $M$ is invertible, which is true if every $lambda_i$ is non-zero). Thus, if $M$ is positive definite, each $lambda_i > 0$, which implies $lambda_i^k > 0$ as well, and therefore $M^k$ is also positive definite [additionally, if $M$ is symmetric, so is $M^k$].

Thus, $v^T M^k v > 0$ for all $v ne 0$, which proves (as shown in the question itself) that the angle between $v$ and $M^k v$ is acute.

We can also see this geometrically. Now I will assume that $M$ is indeed a (real) symmetric $n times n$ positive definite matrix. Being real and symmetric guarantees that it is diagonalisable, or equivalently (what is important for us), that it has a set of $n$ eigenvectors that form a basis for $mathbb R^n$. Indeed, there is an orthonormal basis of $mathbb R^n$ whose elements are eigenvectors of $M$. Let this orthonormal basis be $B = x_1, ldots, x_n$, and let $lambda_i$ be the eigenvalue corresponding to the eigenvector $x_i$, $i = 1, ldots, n$. Thus, $M x_i = lambda_i x_i$.

Thus, given any vector $v in mathbb R^n$, we can decompose it along the basis vectors as, say, $$v = alpha_1 x_1 + cdots + alpha_n x_n.$$

Now, if $M$ is applied to $v$, each component in the above representation gets scaled by the corresponding eigenvalue (because each component is an eigenvector). That is,

beginalign*
Mv &= M(alpha_1 x_1 + cdots + alpha_n x_n)\
&= alpha_1 (M x_1) + cdots + alpha_n (M x_n)\
&= alpha_1 lambda_1 x_1 + cdots + alpha_n lambda_n x_n\
&= lambda_1 (alpha_1 x_1) + cdots + lambda_n (alpha_n x_n).
endalign*

Since each $lambda_i$ is positive, all the components get scaled in their current direction (up, down, or not at all, according to the eigenvalue being greater than, less than, or equal to $1$). This makes it obvious that the direction of none of the components is reversed (and therefore the direction of the whole vector is also not reversed). Furthermore, since no eigenvalue is zero, no "projection" occurs. Thus, $Mv$ cannot be orthogonal to $v$. Indeed, the angle between $Mv$ and $v$ will be acute.

Consider the orthonormal basis $B$ consisting of eigenvectors of $M$. This orthonormal system defines its own $2^n$ orthants in $mathbb R^n$ (not the standard orthants). The vector $v$ lies in one of these, or possibly between some of these (if its components along some of the eigenvectors are zero). If it lies between some orthants, then then we may simply consider the lower dimensional subspace spanned by its non-zero components and all the arguments below will hold in this subspace.

Consider the example shown below in $mathbb R^2$. The light blue vector is $v$ and the shaded region is the orthant containing it in the coordinate system defined by the red axes defined by the two eigenvectors of $M$. The components of $v$ along the axes are also shown in light blue.

Each time $M$ is applied (to the result of the previous application), each component gets scaled by the corresponding eigenvalue. The diagram shows $Mv$ and $M^2 v$ and their respective components (the darker blue vectors). Thus, higher and higher powers of $M$ applied to $v$ produce vectors where the components of $v$ along the eigenvectors corresponding to the highest eigenvalues have been scaled abnormally high (assuming these eigenvalues are greater than $1$). On the other hand, the components corresponding to eigenvalues less than $1$ if any will get scaled down, closer and closer to zero (none such in the example shown).

However, since the scaling never reverses any component, all the vectors from the successive applications remain in the same orthant as the original vector. Any two vectors strictly inside one octant have an acute angle between them.