An 800 by 800 matrix is barely what I would call large anymore. Perhaps 40 years ago, yes. Today, it is almost tiny. Ok, not tiny, perhaps mediocre is a better description by today's standards.
Regardless, if in returns NANs, then the matrix is numerically singular. We can not know if it is truly singular or not without virtually infinte precision arithmetic. Essentially, your matrix has no inverse. So using inv on it returns garbage. What can you expect?
So is your matrix truly singular? This can happen because of a bug in your code, if the matrix was computed incorrectly. Or it could happen because of a problem in your mathematics. For example, it is trivially easy to write engineeering code for the elastic behavior of a truss, but have the matrix be singular/ Thus if I do not force some point in the truss to be fixed in space, then the object can be freely translated to any point in the domain with no cost in potential energy. The matrix decribing the elastic deformation of the trus will be singular. But that is a problem in the mathematics.
And of course, a matrix can be mathematically non-singular, but numerically singular, when the mathematics is performed using floating point arithmetic.
We cannot know which is the case. Did you screw up the code? The mathematics? The numerical linear algebra? Where does the problem lie? Who knows. All we are told is inv did not work.
Worse yet, most of the time when you use inv, you are doing so for the wrong reasons. Just because a formula on a printed page shows a matrix inverse does not necessarily mean you need to use inv.
Usually the backslash operator is a better choice for numerical reasons. And sometimes, if your matrix is singular, then a pseudo-inverse MAY be appropriate. But that too is impossible to know, since only you know what the matrix represents and why you think you need to compute an inverse here.