How to use DAG network - multi-input network (Static + Temporal inputs using LSTM and Fully connected layers)

Hello there,

I'm trying to combine static features with temporal data to affect the outcome of the sequence output with no success yet.

I tried using both R2020a and R2020b to see if there is any difference but, I get the same error mentioned below in both versions.

The problem is a regression/prediction problem (not classification) and both input and output are sequences of unique objects (each row) with different sequence lengths (except the static data). The static data I'm trying to add is at time t = 0 and they are not pictures (each row corresponds to the unique object in each row of XTrain and YTrain).

If anyone can give suggestions or guide me in the right direction, I'll highly appreciate it.

I got the main idea from the paper called "Early Predictions for Medical Crowdfunding: A Deep Learning Approach Using Diverse Inputs" by Wang et al. (2019) which should be publicly available but, unlike them, I'm not trying to initialize the hidden state at time t=0 yet but, trying to use the older approach that they mention at the beginning of the paper. Architecture of one of the older approaches that they mention is below:

Attached is the simplified data saved as excel that is required to run the code.

Everything in the code works until the training part and then I get the following error:

Error using trainNetwork (line 170)
Invalid network.

Error in Static_temporal_NN_Test (line 95)
net = trainNetwork({XTrain(1:3,:);Static_input(1:3,:)'},YTrain(1:3,:),lgraph,options);

Caused by:
    Network: Incompatible layer types. The network contains layer types not supported with recurrent layers.
    Detected recurrent layers:
        layer 'bilstm1' (BiLSTM)
    Detected incompatible layers:
        layer 'static_input' (Image Input)
    Network: Incompatible layer types. The network contains layer types not supported with image input layers.
    Detected image input layers:
        layer 'static_input' (Image Input)
    Detected incompatible layers:
        layer 'flattenstatic1' (Flatten)
    Network: Invalid input layers. If the network has a sequence input layer, then it must not have any other input layers.
    Layer 'concat1': Input size mismatch. Size of input to this layer is different from the expected input size.
    Inputs to this layer:
        from layer 'flattenstatic1' (output size 100)
        from layer 'bilstm1' (output size 100)

Full code below:

clear;
clc;
close all;

%% Import Data



Input = xlsread('Static_temporal_data.xlsx','XTrain','B2:CG4');
Output = xlsread('Static_temporal_data.xlsx','YTrain','B2:CG4');
Static_input = xlsread('Static_temporal_data.xlsx','Static_Features','B2:H4');

XTrain = cell(size(Input,1),1);
YTrain = cell(size(Output,1),1);

for i = 1:size(Input,1)
    
    XTrain{i,1} = Input(i,~isnan(Input(i,:)));
    YTrain{i,1} = Output(i,~isnan(Output(i,:)));
    
end


%% Training Options

miniBatchSize = 1;
maxEpochs = 2;

options = trainingOptions('adam', ...
    'MaxEpochs',maxEpochs,'MiniBatchSize',miniBatchSize,'Shuffle','never', ...
    'GradientThreshold',1, ...
    'InitialLearnRate',0.01, ...
    'LearnRateSchedule','piecewise', ...
    'LearnRateDropPeriod',125, ...
    'LearnRateDropFactor',1, ...
    'Verbose',0, ...
    'Plots','training-progress');

%% BiLSTM Layer(s)


numFeatures = 1;
numHiddenUnits = 50;

% Structure of the LSTM network
layers = [ ...
    sequenceInputLayer(numFeatures, 'Name', 'temporal_input')
    bilstmLayer(numHiddenUnits,'BiasInitializer','ones','OutputMode','sequence', 'Name', 'bilstm1')
    ];



%% Static Layers
numfeatures_static = [7 1];
numResponses_static = 100;

layers_static = [ ...
    imageInputLayer(numfeatures_static,'Name', 'static_input')
    fullyConnectedLayer(numResponses_static,'BiasInitializer','ones', 'Name', 'fullyconnect1_static')
    reluLayer('Name', 'relustatic_1')
    flattenLayer('Name', 'flattenstatic1')
    ];

%% Concatenation Layer + Fully connected Layer + Regression Layer

numResponses = 1;

conclayers = [...
    concatenationLayer(3,2,'Name','concat1')
    fullyConnectedLayer(numResponses,'BiasInitializer','ones', 'Name', 'fullyconnect1_temporal')
    regressionLayer('Name', 'regressionlayer')];

%% Multiple Input DAG NN + Layer Graph plot

lgraph = layerGraph(layers);

plot(lgraph);
axis off;

lgraph = addLayers(lgraph, layers_static);
lgraph = addLayers(lgraph, conclayers);

plot(lgraph);
axis off;

lgraph = connectLayers(lgraph, 'flattenstatic1', 'concat1/in1');
lgraph = connectLayers(lgraph, 'bilstm1', 'concat1/in2');

plot(lgraph);
axis off;


%% Train Network

net = trainNetwork({XTrain(1:3,:);Static_input(1:3,:)'},YTrain(1:3,:),lgraph,options);

[Janez Podobnik]

Any luck with the problem. I am trying something similar, but I also have no luck. It seems that the main error is:

Invalid input layers. If the network has a sequence input layer, then it must not have any other input layers.

As it seems in Matlab you can only have on input if you use sequence input layer.

[Yildirim Kocoglu]

Unfortunately, no luck so far with Matlab. As you said earlier it does not let you use any other type of layer if you use sequence input layer. However, I also saw CNN-LSTM implemented in matlab for videos and this might be useful depending on your application. Let me know, if you find a solution and I will do the same. Good luck.

[Ankita Jain]

Have you find out any solution? I am trying the same. And I guess custom layer option provided my matlab will help to create two input layer.