delay random signal using FFT

This doubt has been pinching me since long. Please help me. I have done the following:

clc
clear all 
d= 0.000875564783; %arbitrary time delay
fs=50000;
T=1/fs;
t=0:T:1-T;
x = sin(2*pi*50*t)+cos(2*pi*100*t); %sinusoid
L=length(x);
e=exp(-1j *2*pi*([0:L/2 -L/2+1:-1])*L*T*d); %delay factor in frequency domain
X=fft(x);
v=X.*e;
out=real(ifft(v));
y=sin(2*pi*50*(t-d))+cos(2*pi*100*(t-d)); %mathematical delay
plot(y-out);

The above code is perfect for delaying all sinusoids and all frequencies (within Nyquist rate).

But what about the real/random signals? What should be the delay factor 'e' in this case? Beacause when I apply it for real signal(recorded speech),(with intension of beamforming) I get misleading results. Also, how can I verify this as I did above in 'y' (mathematically delay)? Given ANY signal x(t), what should be the delay factor?

Please help with a suitable example.