Hi Shafaq,
By directly applying the rotation matrix R_ext to the matrix A and vector b, the mathematical properties are preserved, and the invariance check should now yield results close to zero, indicating that the solution is invariant under rotation. So,applying the rotation matrix R_ext to the matrix A and vector b, we ensure that the mathematical properties are preserved, and the invariance check should now yield results close to zero.
% Extend the rotation matrix to apply to the whole problem R_ext = blkdiag(R, eye(size(A, 1) - 2));
Hope this will help achieve your goal now.