Linearly independence

Definition: Consider a set of n-vectors, {(Subscript[K, 1]^□),(Subscript[K, 2]^□),...,(Subscript[K, n]^□)}. 

If there exist a set of coefficient {(Subscript[x, 1]^□),(Subscript[x, 2]^□),...,(Subscript[x, n]^□)}≠ {0,0,...,0} such that  (Subscript[x, 1]^□)(Subscript[K, 1]^□)+(Subscript[x, 2]^□)(Subscript[K, 2]^□)+...+(Subscript[x, n]^□)(Subscript[K, n]^□)=0, the set of vector {(Subscript[K, 1]^□),(Subscript[K, 2]^□),...,(Subscript[K, n]^□)} is not linearly independent.

In other words, if the only solution for (Subscript[x, 1]^□)(Subscript[K, 1]^□)+(Subscript[x, 2]^□)(Subscript[K, 2]^□)+...+(Subscript[x, n]^□)(Subscript[K, n]^□)=0 is {(Subscript[x, 1]^□),(Subscript[x, 2]^□),...,(Subscript[x, n]^□)}={0,0,...,0}, then the set of vector {(Subscript[K, 1]^□),(Subscript[K, 2]^□),...,(Subscript[K, n]^□)} is linearly independent. 

Example:
Consider the 3-vectors pair, (Subscript[X, 1]^T)=[1,2,3], (Subscript[X, 2]^T)=[-1,-2,-3].
Find any possible values of (Subscript[k, 1]^□) and (Subscript[k, 2]^□), with {(Subscript[k, 1]^□),(Subscript[k, 2]^□)}≠{0,0}, such that 
(Subscript[k, 1]^□)(Subscript[X, 1]^T)+ (Subscript[k, 2]^□)(Subscript[X, 2]^T)=0. (ii) Are the vectors linearly dependent or linearly-independent ?

Ans: (i) Any arbitrary value of (Subscript[k, 1]^□)= (Subscript[k, 2]^□)  ≠ 0 will do. (ii).
They are linearly dependent, since there exist values of  {(Subscript[k, 1]^□),(Subscript[k, 2]^□)}≠{0,0} such that (Subscript[X, 1]^T)= -((Subscript[k, 2]^□)/(Subscript[k, 1]^□))(Subscript[X, 2]^T). Any pairs of vectors that are parallel or antiparallel is not linearly independent. 

Note that when, say for example, the set of 3 vectors {(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□)} are not lienarly independent, any of the vector in the set can be expressed as linear combination of the rest, e.g., 
(Subscript[x, 1]^□)(Subscript[K, 1]^□)+(Subscript[x, 2]^□)(Subscript[K, 2]^□)+(Subscript[x, 3]^□)(Subscript[K, 3]^□)=0
⇒(Subscript[K, 3]^□)=-((Subscript[x, 1]^□)(Subscript[K, 1]^□)+(Subscript[x, 2]^□)(Subscript[K, 2]^□))/(Subscript[x, 3]^□).
In other words, (Subscript[K, 3]^□) is not a vector independent from the rest, (Subscript[K, 2]^□)and (Subscript[K, 1]^□) (because (Subscript[K, 3]^□) can be expressed in terms of (Subscript[K, 1]^□) and (Subscript[K, 2]^□)) 

Example of linearly independent set of vectors
(Subscript[X, 1]^T)=[1,2,3], (Subscript[X, 2]^T)=[4,5,6]. 
The set of two vectors {(Subscript[X, 1]^□),(Subscript[X, 2]^□)} is linearly independent. 

For (Subscript[k, 1]^□)(Subscript[X, 1]^□)+(Subscript[k, 2]^□)(Subscript[X, 2]^□)=(Subscript[[0,0,0], □]^T), we need (Subscript[k, 1]^□) + 4(Subscript[k, 2]^□) = 0, 2(Subscript[k, 1]^□) + 5(Subscript[k, 2]^□) = 0, 3(Subscript[k, 1]^□)+6(Subscript[k, 2]^□)= 0. The only possible solution is (Subscript[k, 1]^□), (Subscript[k, 2]^□) both being zero. This means that for (Subscript[k, 1]^□)(Subscript[X, 1]^□)+(Subscript[k, 2]^□)(Subscript[X, 2]^□)=(Subscript[[0,0,0], □]^T), the coefficients {(Subscript[k, 1]^□), (Subscript[k, 2]^□)} must be all zero. This proves the linearly independence of (Subscript[X, 1]^□) and (Subscript[X, 1]^□).


Refer (9.5) in page 69, Ayres. Given a set of m  vectors, we want to know whether they are linearly independent or otherwise. What is the easiest way (or one of the easier ways) to determine the linear independence of such a set of vectors? 

Ans: Use row elementary operations to reduce the matrix A formed by these vectors to RREF. The number of non-zero row in the RREF of A is the rank of the matrix A, r. The rank, r, also tells us how many linearly independent vectors are there in the set of m vectors. 

If r = m, then the set of this m vectors is linearly independent. If r < m, then the set of m vectors is linearly dependent. 

Example 2.3, Prof. Rosy Teh’s note on Linear Algebra, page 33.
Consider the set S containing the following 4 3-vectors: (Subscript[K, 1]^□)=(([1,1,1])^T), (Subscript[K, 2]^□)=(([1,3,5])^T), (Subscript[K, 3]^□)=(([1,5,3])^T),(Subscript[K, 4]^□)=(([5,3,1])^T);
S={(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□),(Subscript[K, 4]^□)}. (i) Form the matrix K whose rows are made up of the vectors (Subscript[K, i]^□), i=1, 2, 3, 4. (ii) Reduce K into RREF. (iii) What is the rank of K? (iv) How many linearly independent vectors are there in the set S? (v) Are the vectors in set S linearly independent?

Answer:
K=((Subscript[K, 1]^□) (Subscript[K, 2]^□) (Subscript[K, 3]^□) (Subscript[K, 4]^□))=(({{1, 1, 1, 5}, {1, 3, 5, 3}, {1, 5, 3, 1}})). 
RREF(K)=(({{1, 0, 0, 6}, {0, 1, 0, -1}, {0, 0, 1, 0}})). Obviously, the rank of matrix K is r=3. 
Since the number of vectors in S, m = 4 is larger than the number of linearly independent vectors, r = 3, the set of vectors in S is not linearly independent. They are linearly dependent, by the virtue of theorem V, page 69, Ayers.

(* another example *)
K1={1,1,1};K2={1,3,5};K3={1,5,3};
A={{1,1,1},{1,3,5},
{1,5,3}};
Print["K1=",K1//MatrixForm, ", K2=",K2//MatrixForm, ", K3=",K3//MatrixForm];
Print["A = (K1 K2 K3)=",A//MatrixForm];
Print["m = number of vectors forming the matrix A = ",Length[A]];
Print["RREF[A] = ",RowReduce[A]//MatrixForm];
Print["Hence Rank [A] is r = ",MatrixRank[A]];
Print["Since r = m = ",MatrixRank[A],"the set of vector A is linearly independent"];

(*example 2.2, page 33, Prof. Rosy Teh's lecture note on Linear Algebra *)
K1={1,1,1};
K2={1,3,5};
K3={1,5,3};
K4={5,3,1};
K={{1,1,1,5},{1,3,5,3},
{1,5,3,1}};
Print["K1=",K1//MatrixForm, ", K2=",K2//MatrixForm, ", K3=",K3//MatrixForm,", K4=",K4//MatrixForm];
Print["K = (K1 K2 K3 K4)=",K//MatrixForm];

H={a,b,c};
X={x1,x2,x3,x4};
aug={{1,1,1,5,a},{1,3,5,3,b},
{1,5,3,1,c}};
Print["H is ",H // MatrixForm ];
Print["X is ",X // MatrixForm ];
Print["K X = H =  "];
Print[K//MatrixForm, X// MatrixForm,"=",K.X// MatrixForm,"=",H //MatrixForm ];
Print["Rank [K]=",MatrixRank[K]];
Print["Rank [K|H]=",MatrixRank[aug]];
Print["Hence the non-homegeneous system K X = H is consistent, i.e., there are solution such that X ≠",{0,0,0,0}//MatrixForm];
Print["A .X =",K // MatrixForm,X // MatrixForm,"=",{a,b,c}// MatrixForm ];
Print["RREF[K].X =",RowReduce[K ] // MatrixForm,X // MatrixForm,"=",RowReduce[K ].X// MatrixForm,"=",MatrixForm[{a,b,c}]];
Print["The exact values of x1,x2,x3,x4 are not important here. What is important is that the non-trivial solution, X, exist, and by definition, the set of vectors {K1, K2, K3, K4} spans S=V_3(R). In other words, the set of vectors {K1, K2, K3, K4} is said to span the vector space V_3(R) because  
any arbitrary vector H in V_3(R) can be expressed as linear combinations of {K1, K2, K3, K4} with coefficients {x1,x2,x3,x4} that are not all zero."]

In the example of 2.2, the set of vectors S={(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□),(Subscript[K, 4]^□)} is proven to span (Subscript[V, 3]^□)(R). We ask another different question:  does the set of vector R={(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□)} span (Subscript[V, 3]^□)(R)?
To answer the question:

K1={1,1,1};
K2={1,3,5};
K3={1,5,3};
K={{1,1,1},{1,3,5},
{1,5,3}};
Print["K1=",K1//MatrixForm, ", K2=",K2//MatrixForm, ", K3=",K3//MatrixForm];
Print["K = (K1 K2 K3)=",K//MatrixForm];

H={a,b,c};
X={x1,x2,x3};
aug={{1,1,1,a},{1,3,5,b},
{1,5,3,c}};
Print["H is ",H // MatrixForm ];
Print["X is ",X // MatrixForm ];
Print["K X = H =  "];
Print[K//MatrixForm, X// MatrixForm,"=",K.X// MatrixForm,"=",H //MatrixForm ];
Print["Rank [K]=",MatrixRank[K]];
Print["Rank [K|H]=",MatrixRank[aug]];
Print["Hence the non-homegeneous system K X = H is consistent"];
Print["There is non-trivial solution X=[x1,x2,x3]^T such that any arbitray vector H can be expressed as linear combination of K1,K2 and K3, H = K.X. By definition, the set of vectors {K1, K2, K3} spans S=V_3(R)."]

In the example above, the set of vectors S={(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□)} is proven to span (Subscript[V, 3]^□)(R). We ask another different question:  does the set of vector R={(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 4]^□)} span (Subscript[V, 3]^□)(R)?
To answer the question:

K1={1,1,1};
K2={1,3,5};
K4={5,3,1};
K={{1,1,5},{1,3,3},
{1,5,1}};
Print["K1=",K1//MatrixForm, ", K2=",K2//MatrixForm, ", K4=",K4//MatrixForm];
Print["K = (K1 K2 K4)=",K//MatrixForm];

H={a,b,c};
X={x1,x2,x3};
aug={{1,1,5,a},{1,3,3,b},
{1,5,1,c}};
Print["H is ",H // MatrixForm ];
Print["X is ",X // MatrixForm ];
Print["K X = H =  "];
Print[K//MatrixForm, X// MatrixForm,"=",K.X// MatrixForm,"=",H //MatrixForm ];
Print["Rank [K]=",MatrixRank[K]];
Print["Rank [K|H]=",MatrixRank[aug]];
Print["Hence the non-homegeneous system KX= H is NOT consistent."];
Print["There is not solution X=[x1,x2,x3]^T such that any arbitray vector H can be expressed as linear combination of K1,K2 and K3, H = K.X. By definition, the set of vectors {K1, K2, K4} does not spans S=V_3(R)."]

We ask another different question:  does the set of vector R={(Subscript[K, 1]^□),(Subscript[K, 2]^□)} span (Subscript[V, 3]^□)(R)? To answer this question:

K1={1,1,1};
K2={1,3,5};
K={{1,1},{1,3},
{1,5}};
Print["K1=",K1//MatrixForm, ", K2=",K2//MatrixForm];
Print["K = (K1 K2)=",K//MatrixForm];

H={a,b,c};
X={x1,x2};
aug={{1,1,a},{1,3,b},
{1,5,c}};
Print["H is ",H // MatrixForm ];
Print["X is ",X // MatrixForm ];
Print["K X = H =  "];
Print[K//MatrixForm, X// MatrixForm,"=",K.X// MatrixForm,"=",H //MatrixForm ];
Print["Rank [K]=",MatrixRank[K]];
Print["Rank [K|H]=",MatrixRank[aug]];
Print["Hence the non-homegeneous system K X = H is NOT consistent, i.e., there are no non-trivial solution for K X = H "];
Print["It is not possible to expressed H in terms of linear combination of K1, K2 in S=V_3(R). By definition, the set of vector {K1, K2} does not span V_3(R)"]

We ask another different question:  does the set of vector R={(Subscript[K, 1]^□),(Subscript[K, 2]^□)} linearly independent?

K1={1,1,1};K2={1,3,5};
A={{1,1},{1,3},
{1,5}};
Print["K1=",K1//MatrixForm, ", K2=",K2//MatrixForm];
Print["A = (K1 K2 )=",A//MatrixForm];
Print["m = number of vectors forming the matrix A = ",Length[A]];
Print["RREF[A] = ",RowReduce[A]//MatrixForm];
Print["Hence Rank [A] is r = ",MatrixRank[A]];
Print["Since r = m = ",MatrixRank[A],"the set of vector A is linearly independent"];

Conclusion: 
{(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□),(Subscript[K, 4]^□)} span (Subscript[V, 3]^□)(R), and is not linearly independent.
{(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□)} span (Subscript[V, 3]^□)(R), and is linearly independent.
{(Subscript[K, 1]^□),(Subscript[K, 2]^□)} does not span (Subscript[V, 3]^□)(R), and is linearly independent.

{(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□)} are basis vectors in (Subscript[V, 3]^□)(R), as any abitrary vector H in  (Subscript[V, 3]^□)(R) can be expressed as linear combination of {(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□),(Subscript[K, 4]^□)} or {(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□)}.  There are many other set of basis vectors in  (Subscript[V, 3]^□)(R), other than {(Subscript[K, 1]^□),(Subscript[K, 2]^□),(Subscript[K, 3]^□)}, as long as the set of vector is linearly independent and span (Subscript[V, 3]^□)(R). For example, the set of vectors {(([1,0,0])^T), (([0,1,0])^T), (([0,0,1])^T)} is an good example of basis vectors 

The minimum number of linearly independent vector to span a space is the dimension of the vector space. In the above example, the vector space (Subscript[V, 3]^□)(R) has a dimension of 3, because that is the minimum of linearly independent vectors that is required to span it.