How to create a positional encoder layer using deep network designer?

Im creating the network architecture of my transformer but I don't know how to put the positional information "positional encoding" with Deep Network Designer, this is a code that I have so far...

function lgraph = setNeuralNetworkArchitecture(inputSize, numClasses)

% Create layer graph

lgraph = layerGraph();

% Add layer branches

tempLayers = [

sequenceInputLayer([13 24 8],"Name","sequence")

% Añadir positional encoding

lambdaLayer(@(x) addPositionalEncoding(x, encodingDim), 'Name', 'positionalEncoding')

flattenLayer("Name","flatten")];

lgraph = addLayers(lgraph,tempLayers);

tempLayers = selfAttentionLayer(16,64,"Name","selfattention");

lgraph = addLayers(lgraph,tempLayers);

tempLayers = [

additionLayer(2,"Name","addition")

fullyConnectedLayer(6,"Name","fc_1")

softmaxLayer("Name","softmax")

classificationLayer("Name","classoutput")];

lgraph = addLayers(lgraph,tempLayers);

% clean up helper variable

clear tempLayers;

lgraph = connectLayers(lgraph,"positionalEncoding","flatten");

lgraph = connectLayers(lgraph,"flatten","selfattention");

lgraph = connectLayers(lgraph,"flatten","addition/in2");

lgraph = connectLayers(lgraph,"selfattention","addition/in1");

end

Answer 1

[Siraj]

Hii! @Luis Gabriel Macias Santillan It is my understanding that you want to add a positional encoder layer using the deep network designer app available in MATLAB. According to my research, the deep network designer doesn’t have a positional encoder layer that could be directly added to the network.

You can refer to the following documentation to understand how we can integrate a custom deep-learning layer.

Define Custom Deep Learning Intermediate Layers - MATLAB & Simulink - MathWorks India

Also, you can check out the following file, which has an implementation for a positional encoder.

wav2vec-2.0 - File Exchange - MATLAB Central (mathworks.com)

Hope this helps.