I assume the elements of x and y are linked? Why is the first element of (x2,y2) than not in the list of coordinaties (x1,y1)? I assume this is a mistake.
If I were you, I would link the coordinates explicitly, as rows of a N-by-2 array, allowing to use SETDIFF:
xy_all = [x1(:) y1(:)]
xy_nonzero = [x2(:) y2(:)]
xy_zero = setdiff(xy_nonzero, xy_all, 'rows')