streaming real-time financial data in MATLAB

Hi all, I would like some guidance as to how to frame the following streaming problem in MATLAB. I am currently trying to stream (real time) financial data, process it with some user-defined functions, and plot then result. The data arrives with a frequency of 10 obs per second (although it could be a higher frequency). As the data is received, I accumulate it in a struct with a persistent variable. Right now, one background pool receives the streaming data and accumulates it in a struct. I use AfterEach to process the observations as they are received, and plot the result. Here's a brief (simplified) illustration:

p = parpool; 
queue1 = parallel.pool.DataQueue;
queue1.afterEach(@(x) processData(x)); 
f = parfeval(@genData,0,queue1); 
function genData(queue1)
len = 1000; 
for jj = 1:len 
    send(queue1, randn) 
end 
end 

function processData(x) 
persistent y iter; 
if isempty(y) 
    iter = 0; 
    y = nan(1000,1); % i tried to preallocate in some way 
    iter = iter +1; 
else 
    y(iter) = x; 
    iter = iter + 1; 
end
end
    % do stuff with y
    % call user defined functions with y as input
    % plot y and so on... things that might take longer than 0.1 seconds
    % fft(y(1:iter-1));
    Z = anotherFunc(y(1:iter));
    plot(z);
end

The execution of processData takes longer and longer every time, and at some point during the day the execution time is longer than the time it takes to receive two consecutive observations. (Maybe running the code above might not illustrate this problem because it is too simplified). I assume this happens because functions that I'm calling take

longer and longer each time. I can continue down this path, trying to manage the memory, or somehow only keep the most recent observations and only process those and so on, but maybe matlab already has something that might help me wield this stream of data easier.

My question is the following: what does Matlab offer to process realtime data streaming as above? Could you please give me a simplified example that might give me a headstart ? I ran into the following description and some listed toolboxes (audio, computer vision...) but they seem to be specialized to particular applications: https://www.mathworks.com/discovery/stream-processing.html

Answer 1

[Shivam Lahoti]

Hi dleal,

I understand that you want to leverage MATLAB and process real-time financial data without implementing any efficiency techniques in your provided code.

Please have a look at the Datafeed Toolbox that MATLAB provides. You can establish connections from MATLAB to retrieve historical, intraday, or real-time data streams and then perform analyses, develop models and financial trading strategies, and create visualizations that reflect financial and market behavior.

https://www.mathworks.com/products/datafeed.html

Datafeed Toolbox™ functions enable you to create a FactSet Workstation connection and retrieve real-time data.

First, create a FactSet Workstation connection by using the ‘fds’ function, and then use the ‘realtime’ function. Please refer to the example below to understand how these functions are used and then you can refer to other functions according to your use-case.

username = 'ABCD_EFGH_IJKL';
password = 'XXXXXXXX';
c = fds(username,password)
Srv = 'FDS1';
Sec = 'ABCD-USA';
Cb = @(varargin)myMessageEventHandler(varargin);
t = realtime(c,Srv,Sec,Cb)

I hope this was helpful.

Regards,

Shivam Lahoti.