My understanding is that you would like to rotate this given plane (which passes through origin) such that the normal of the plane is parallel to z-axis. And this is what you mean by flattening the plane.
1. I don't understand why the sequence of rotation mentioned in the question would result into flattening of this plane.
2. The final orientation of the plane will depend on the sequence and the angle of rotations with which the plane is rotated about the axes.
3. First we need to find the normal of the original plane. This can be easily done by taking the cross product of any three linearly independent points on the plane. Here the unit normal (un) is 1/root(3)*[1 1 1]. You can also take negative of this normal.
4. After the rotation of the plane our desired unit normal (vn) should be [0 0 1]. You can also take negative of this normal.
5. The axis of rotation about which we rotate this plane is the cross product of un and vn. And the angle of rotation is the cosine inverse of the inner dot product of un and vn.
6. With the help of this compute the rotation matrix using the MATLAB function ' vrrotvec2mat ' or compute it using the equations described here .
7. Finally use this rotation matrix to get the desired result.
8. To get the Eulerian angles use the function 'rotm2eul' or again compute it using other resources from Google like this .
9. The Eulerian angles are -15 degree about Z axis, -35.2644 degree about Y axis and 45 degree about X axis. The sequence of rotation is ZYX. You can use these angles to verify this at your end.
Please refer to the code below for illustration purpose. Please note this requires access to the following functions: a. vrrotvec2mat - requires Simulink 3D Animation Toolbox b. rotm2eul - requires Robotics System Toolbox
p = @(x) [X(x),Y(x),Z(x)];
rotX = @(x) [1 0 0; 0 cosd(x) -sind(x); 0 sind(x) cosd(x)];
rotY = @(x) [cosd(x) 0 sind(x);0 1 0; -sind(x) 0 cosd(x)];
rotZ = @(x) [cosd(x) -sind(x) 0; sind(x) cosd(x) 0; 0 0 1];
u = cross(p(3)-p(11),p(3)-p(16));
un = u'/norm(u,2);
vn = [0;0;1];
rot = vrrotvec2mat([cross(un,vn); acos(un'*vn)]);
newX_t = X*rot(1,1)+Y*rot(1,2)+Z*rot(1,3);
newY_t = X*rot(2,1)+Y*rot(2,2)+Z*rot(2,3);
newZ_t = X*rot(3,1)+Y*rot(3,2)+Z*rot(3,3);
eul = rotm2eul(rot,'ZYX')*180/pi;
rot_t = rotZ(eul(1))*rotY(eul(2))*rotX(eul(3));