i want to use LSTM based audio network to work with Live audio
Show older comments
Hello Matlab team,
I am using this example to work with my audio data set https://www.mathworks.com/matlabcentral/fileexchange/74611-fault-detection-using-deep-learning-classification#examples_tab dataset is trained but I want to make the application live with PC, forexample I have a mic and make an application to use my trained model to predict the output.
Can you guide me or help me with that?
Regards,
Arslan Munaim
Answers (2)
jibrahim
on 27 Jul 2022
Hi Arslan,
There is a function in that repo (streamingClassifier) that should get the job done in conjunction with an audio device reader:
% Create a microphone object
adr = audioDeviceReader(SampleRate=16e3,SamplesPerFrame=512);
% These statistic value should come from your training...
M = 0;
S = 1;
while 1
% Read a frame of data from microphone
frame = adr();
% Pass to network
scores = streamingClassifier(frame,M,S);
% Use the scores any way you want
end
5 Comments
Arslan Munim
on 28 Jul 2022
Edited: Arslan Munim
on 28 Jul 2022
jibrahim
on 28 Jul 2022
Hi Arslan,
The extract function should also return 11 features. For example, if you replace the eixsting function extractFeatures with this modified function, things should work the same:
function featureVector = extractFeatures2(x)
%#codegen
persistent afe
if isempty(afe)
windowLength = 512;
overlapLength = 0;
afe = audioFeatureExtractor('SampleRate',44100, ...
'Window',hamming(windowLength,'periodic'),...
'OverlapLength',overlapLength,...
'spectralCentroid',true, ...
'spectralCrest',true,...
'spectralDecrease',true, ...
'spectralEntropy',true,...
'spectralFlatness',true,...
'spectralFlux',true,...
'spectralKurtosis',true,...
'spectralRolloffPoint',true,...
'spectralSkewness',true,...
'spectralSlope',true,...
'spectralSpread',true);
end
featureVector = extract(afe,x);
end
The size of featureVector will be 1-by-11, each element in the vector representing one of your features.
Notice I declared afe as persistent. This is to ensure the audio feature extractor is not recreated every time you call this function in your loop. the extractor goes through some one-time setup computations when you first call it. No need to waste time repeating those.
Arslan Munim
on 2 Aug 2022
jibrahim
on 2 Aug 2022
Hi Arslan,
Since you trained the network with a sample rate of 16e3, you will have to perform sample-rate conversion from 44100 kHz to 16 kHz. This code is a possible implementation, where you essentially feed the network frames of length 512 sampled at 16 kHz, just like the original code:
% Create a microphone object
%adr = audioDeviceReader(SampleRate=16e3,SamplesPerFrame=512);
src = dsp.SampleRateConverter(InputSampleRate=44100,OutputSampleRate=16e3,...
Bandwidth=15800);
[~,D] = src.getRateChangeFactors;
% The frame size must be a multiple of 441 (the decimation factor of the
% sample rate converter)
L = floor(22000/D);
frameLength = L*D; % get as close to desired frame size
adr = audioDeviceReader(SampleRate=44100,SamplesPerFrame=frameLength);
buff = dsp.AsyncBuffer;
% These statistic values should come from your training...
M = 0;
S = 1;
while 1
% Read a frame of data from microphone
frame = adr();
% Convert to 16 KHz
frame = src(frame);
% Save to buffer
write(buff,frame)
while buff.NumUnreadSamples >= 512
frame = read(buff,512);
% Pass to network
scores = streamingClassifier(frame,M,S);
% Use the scores any way you want
end
end
Note that you can also potentially feed the network longer frames. That should also work, and is probably more efficient, as the network will run faster if you give it a long input (as opposed to multiple short ones):
% Create a microphone object
%adr = audioDeviceReader(SampleRate=16e3,SamplesPerFrame=512);
src = dsp.SampleRateConverter(InputSampleRate=44100,OutputSampleRate=16e3,Bandwidth=15800);
[~,D] = src.getRateChangeFactors;
% The frame size must be a multiple of 441 (the decimation factor of the
% sample rate converter)
L = floor(22000/D);
frameLength = L*D;
adr = audioDeviceReader(SampleRate=44100,SamplesPerFrame=frameLength);
buff = dsp.AsyncBuffer;
% These statistic values should come from your training...
M = 0;
S = 1;
while 1
% Read a frame of data from microphone
frame = adr();
% Convert to 16 KHz
frame = src(frame);
% Save to buffer
write(buff,frame)
N = buff.NumUnreadSamples;
L = floor(N/512);
if L>0
frame = read(buff,512*L);
% Pass to network
scores = streamingClassifier(frame,M,S);
% Use the scores any way you want
end
end
If you can't change the frame size on the microphone, then you can handle that using another buffer, for example:
% Create a microphone object
%adr = audioDeviceReader(SampleRate=16e3,SamplesPerFrame=512);
src = dsp.SampleRateConverter(InputSampleRate=44100,OutputSampleRate=16e3,Bandwidth=15800);
[~,D] = src.getRateChangeFactors;
% The frame size must be a multiple of 441 (the decimation factor of the
% sample rate converter)
L = floor(22000/D);
frameLength = L*D;
adr = audioDeviceReader(SampleRate=44100,SamplesPerFrame=22000);
buffSRC = dsp.AsyncBuffer;
buff = dsp.AsyncBuffer;
% These statistic values should come from your training...
M = 0;
S = 1;
while 1
% Read a frame of data from microphone
frame = adr();
write(buffSRC,frame);
frame = read(buffSRC,frameLength);
% Convert to 16 KHz
frame = src(frame);
% Save to buffer
write(buff,frame)
N = buff.NumUnreadSamples;
L = floor(N/512);
if L>0
frame = read(buff,512*L);
% Pass to network
scores = streamingClassifier(frame,M,S);
% Use the scores any way you want
end
end
Arslan Munim
on 9 Aug 2022
jibrahim
on 9 Aug 2022
0 votes
Hi Arslan,
audioDeviceReader supports multi-mic devices. Use the ChannelMappingSource and ChannelMapping properties to map between device input channels and the output data.
This network was trained on mono data, so, to adapt it to multi-channel data, you either have to retrain your network for multi-channel data, or somehow combine your input channels into one channel (by a weighted sum, or selecting a particular channel, etc) and proceed like above.
23 Comments
Arslan Munim
on 9 Aug 2022
Edited: Arslan Munim
on 9 Aug 2022
jibrahim
on 9 Aug 2022
Yes, I think this is possible. For example, here is how you do two predictions on two independent sets of features:
[airCompNet,scores] = predictAndUpdateState(airCompNet,{randn(10,1),randn(10,1)})
So, you can extract features from each channel, and then get scores for each channel
Arslan Munim
on 9 Aug 2022
jibrahim
on 9 Aug 2022
Yep!
Arslan Munim
on 17 Aug 2022
Edited: Walter Roberson
on 19 Aug 2022
jibrahim
on 17 Aug 2022
You should not create two audio device readers. It seems like you are reading from the same (multichannel) device.Create one audioDeviceReader, and call it. It will return the output of each mic as a separate channel. Use the ChannelMappingSource and ChannelMapping properties to control which output channel corresponds to what microphone.
Arslan Munim
on 17 Aug 2022
jibrahim
on 17 Aug 2022
Hi Arslan,
You can only create one audioDeviceReader at a time. You can't use multiple ones, so we support devices that return multiple channels. I suggest you create one object with either device name, and call it and see what you get back (how many channels?)
Arslan Munim
on 17 Aug 2022
Jimmy Lapierre
on 17 Aug 2022
Hi Arslan, just to clarify, do you have one USB sound card with several mics hooked up to it, or several USB microphones?
Arslan Munim
on 17 Aug 2022
Edited: Arslan Munim
on 17 Aug 2022
jibrahim
on 19 Aug 2022
Arslan, we support the scenario with one USB card with several mics hooked to it. You can't use audioDeviceReader to read from separate cards at the same time. Even if we did, since these different mics run on different clocks, I am not sure how you would achieve synchronization between them anyway.
One possible workaround is to use a different MATLAB session to read from the other microphone, and send the data to MATLAB via UDP. So, in another MATLAB, run some code like this:
sender = dsp.UDPSender(RemoteIPPort=25000);
src = audioDeviceReader;
while(1)
frame = src();
sender(frame);
end
Them, in the main MATLAB, you can receive the audio:
rec = dsp.UDPReceiver(LocalIPPort=25000);
scope = timescope;
while(1)
frame = rec();
scope(frame);
end
This might work if your sound is in steady state and does not change often/fast. If synchronization between mics becomes an issue, then I think one card with multiple devices associated with it is definitely the way to go.
Arslan Munim
on 19 Aug 2022
jibrahim
on 19 Aug 2022
There should be one driver that aggregates the mics, so this will probably not work.
What device(s) are you working with?
Arslan Munim
on 19 Aug 2022
jibrahim
on 19 Aug 2022
If that is the case, then you should be able to open the aggregating device only once (with one audioDeviceReader) as a multichannel device. You should find out the name of the aggregating device and choose that one.
Arslan Munim
on 19 Aug 2022
jibrahim
on 19 Aug 2022
Perhaps this helps:
devinfo = audiodevinfo
See if there is another recongized device name you can use. My guess is that one name corresponds to the single USB card
Arslan Munim
on 19 Aug 2022
jibrahim
on 19 Aug 2022
I am assuming you checked all the inputs (IDs 1, 2. ...etc). If each one corresponds to just one mic, then it would mean that MATLAB does not recognize the aggregating device. Check if your system (outside MATLAB) recongizes the aggregating device or not.
Arslan Munim
on 19 Aug 2022
jibrahim
on 20 Aug 2022
OK, this helps. You will need other hardware (one device, multiple mics) for the system to recognize it. You could also give the UDP idea a shot, see how viable that is.
Arslan Munim
on 28 Sep 2022
Categories
Find more on Audio Processing in Help Center and File Exchange
on 27 Jul 2022
on 28 Sep 2022
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
