You are now following this question
- You will see updates in your followed content feed.
- You may receive emails, depending on your communication preferences.
Is there any layer defined in matlab for sine activation function? Or else can we define the layer using functionLayer?
7 views (last 30 days)
Show older comments
How we can use a layer with sine activation function? Is it possible to use the functionLayer to define a sine activation function or should I define a class for creating the layer as shown in https://in.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-intermediate-layers.html?
Accepted Answer
Matt J
on 1 Dec 2023
If it will not have any learnable parameters, you can use a functionLayer.
8 Comments
BIPIN SAMUEL
on 8 Dec 2023
I be created a customized layer based on attention mechanism for deep learning application using https://in.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layer.html. But while checking the validity of layer using the following code:
layer=CoAtten(Name="atten");
validInputSize = [1 14 1024];
layout = networkDataLayout(validInputSize,"CBT");
layer = initialize(layer,layout);
checkLayer(layer,validInputSize,ObservationDimension=3)
The last line (checkLayer) shows the error like this- ""The value of 'Layer' is invalid. Layers that require formatted dlarray inputs are not supported."
What this error shows? Is the custom layer is invalid? I have used stripdims inside the predict fuction while creating the layer. Is it happening due to this?
Matt J
on 8 Dec 2023
And why ObservationDimension=3 when you have a Batch dimension that is the second dimension?
BIPIN SAMUEL
on 8 Dec 2023
Thank You @Matt J, I have used checkLayer from https://in.mathworks.com/help/deeplearning/ref/checklayer.html for the checking the validity of the layer. I just want to know whether the layer that I have created is correct or not. I have given ObservationDimension=3 to show the size of dlarray input but I am not sure whether it is correct or not.
While using that command "The value of 'Layer' is invalid. Layers that require formatted dlarray inputs are not supported." - this error is showing, but I am not sure that what it represents whether the error is with the layer or with the way I have used the checkLayer(layer,layout,ObservationDimension=3) command. Is there any other way to check the customized layer is working or not?
BIPIN SAMUEL
on 8 Dec 2023
Edited: Matt J
on 8 Dec 2023
classdef CoAtten < nnet.layer.Layer ...
& nnet.layer.Formattable ...
% & nnet.layer.Acceleratable
% properties
% % (Optional) Layer properties.
%
% % Declare layer properties here.
% end
properties (Learnable)
% (Optional) Layer learnable parameters.
alpha
end
% properties (State)
% % (Optional) Layer state parameters.
%
% % Declare state parameters here.
% end
%
% properties (Learnable, State)
% % (Optional) Nested dlnetwork objects with both learnable
% % parameters and state parameters.
%
% % Declare nested networks with learnable and state parameters here.
% end
methods
function layer = CoAtten(NameValueArgs)
% (Optional) Create a myLayer.
% This function must have the same name as the class.
arguments
NameValueArgs.Name = '';
end
name=NameValueArgs.Name;
layer.Name=name;
layer.Description="Attention Mechanism based on correlation";
layer.Type="Correlation Attention";
end
function layer = initialize(layer,layout)
% (Optional) Initialize layer learnable and state parameters.
if isempty(layer.alpha)
idx=finddim(layout,"C");
numChannels = layout.Size(idx);
layer.alpha=dlarray(zeros(numChannels));
end
end
function Z = predict(layer,varargin)
% Forward input data through the layer at prediction time and
% output the result and updated state.
%
X=varargin;
filtersize=ones(1);
nc=ones(1);
numfilters=ones(1);
sz=[filtersize,nc,numfilters];
numout=prod(filtersize)*numfilters;
numin=prod(filtersize)*numfilters;
weights=initializeGlorot(sz,numout,numin);
bias=dlarray(zeros(nc));
query=dlconv(X,weights,bias,WeightsFormat='CUT');
weights=initializeGlorot(sz,numout,numin);
key=dlconv(X,weights,bias,WeightsFormat='CUT');
weights=initializeGlorot(sz,numout,numin);
value=dlconv(X,weights,bias,WeightsFormat='CUT');
X=stripdims(X);
query=stripdims(query);
key=stripdims(key);
value=stripdims(value);
X_zscore=zscore(X);
X_transpose=permute(X_zscore,[1 3 2]);
X_flatten=permute(X_transpose,[2 3 1]);
query_flatten=permute(query,[2 4 1 3]);
query_energy=pagemtimes(X_flatten,query_flatten);
query_energy=permute(query_energy, [3 1 4 2]);
key_flatten=permute(key,[2 4 1 3]);
key_energy=pagemtimes(X_flatten,key_flatten);
key_energy=permute(key_energy,[3 1 4 2]);
query_energy=permute(query_energy,[2 3 1]);
key_energy=permute(key_energy,[2 4 1 3]);
energy=pagemtimes(query_energy,key_energy);
energy=permute(energy,[3 1 4 2]);
energy=permute(energy, [2 1 3]);
attention=softmax(energy,'DataFormat',"UCU");
value_flatten=permute(value,[2 3 1]);
out=pagemtimes(value_flatten,attention);
out=permute(out,[2 1 3]);
Alpha=layer.alpha;
Z=Alpha*out+X;
Z=dlarray(Z,"CBT");
end
end
end
This is the layer that I have used for the validity check using "checkLayer".
Matt J
on 8 Dec 2023
Does this make any more sense?
layer=CoAtten(Name="atten");
validInputSize = [1 14 1024];
layout = networkDataLayout(validInputSize,"CBT");
layer = initialize(layer,layout);
checkLayer(layer,layout,ObservationDimension=2)
Skipping GPU tests. No compatible GPU device found.
Skipping code generation compatibility tests. To check validity of the layer for code generation, specify the CheckCodegenCompatibility and ObservationDimension options.
Running nnet.checklayer.TestLayerWithoutBackward
..
================================================================================
nnet.checklayer.TestLayerWithoutBackward/formattableLayerPredictIsFormatted(Observations=one) was filtered.
Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
.
================================================================================
nnet.checklayer.TestLayerWithoutBackward/formattableLayerPredictIsFormatted(Observations=multiple) was filtered.
Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
....
================================================================================
Verification failed in nnet.checklayer.TestLayerWithoutBackward/predictDoesNotError(Observations=one).
----------------
Test Diagnostic:
----------------
Test failure may be due to the layer not being initialized. If the layer is not initialized, then initialize it by calling its initialize method.
---------------------
Framework Diagnostic:
---------------------
The function 'predict' threw an error:
Undefined function 'initializeGlorot' for input arguments of type 'double'.
Error in CoAtten/predict (line 57)
weights=initializeGlorot(sz,numout,numin);
------------------
Stack Information:
------------------
In /MATLAB/toolbox/nnet/cnn/+nnet/+checklayer/TestLayerWithoutBackward.m (TestLayerWithoutBackward.predictDoesNotError) at 26
================================================================================
.
================================================================================
Verification failed in nnet.checklayer.TestLayerWithoutBackward/predictDoesNotError(Observations=multiple).
----------------
Test Diagnostic:
----------------
Test failure may be due to the layer not being initialized. If the layer is not initialized, then initialize it by calling its initialize method.
---------------------
Framework Diagnostic:
---------------------
The function 'predict' threw an error:
Undefined function 'initializeGlorot' for input arguments of type 'double'.
Error in CoAtten/predict (line 57)
weights=initializeGlorot(sz,numout,numin);
------------------
Stack Information:
------------------
In /MATLAB/toolbox/nnet/cnn/+nnet/+checklayer/TestLayerWithoutBackward.m (TestLayerWithoutBackward.predictDoesNotError) at 26
================================================================================
.. ....
================================================================================
nnet.checklayer.TestLayerWithoutBackward/predictIsConsistentInType(Precision=single,Device=cpu) was filtered.
Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
.
================================================================================
nnet.checklayer.TestLayerWithoutBackward/predictIsConsistentInType(Precision=double,Device=cpu) was filtered.
Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
..... ..
================================================================================
nnet.checklayer.TestLayerWithoutBackward/backwardPropagationDoesNotError(Observations=one) was filtered.
Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
.
================================================================================
nnet.checklayer.TestLayerWithoutBackward/backwardPropagationDoesNotError(Observations=multiple) was filtered.
Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
.
Done nnet.checklayer.TestLayerWithoutBackward
__________
Failure Summary:
Name Failed Incomplete Reason(s)
=================================================================================================================================================
nnet.checklayer.TestLayerWithoutBackward/formattableLayerPredictIsFormatted(Observations=one) X Filtered by assumption.
-------------------------------------------------------------------------------------------------------------------------------------------------
nnet.checklayer.TestLayerWithoutBackward/formattableLayerPredictIsFormatted(Observations=multiple) X Filtered by assumption.
-------------------------------------------------------------------------------------------------------------------------------------------------
nnet.checklayer.TestLayerWithoutBackward/predictDoesNotError(Observations=one) X Failed by verification.
-------------------------------------------------------------------------------------------------------------------------------------------------
nnet.checklayer.TestLayerWithoutBackward/predictDoesNotError(Observations=multiple) X Failed by verification.
-------------------------------------------------------------------------------------------------------------------------------------------------
nnet.checklayer.TestLayerWithoutBackward/predictIsConsistentInType(Precision=single,Device=cpu) X Filtered by assumption.
-------------------------------------------------------------------------------------------------------------------------------------------------
nnet.checklayer.TestLayerWithoutBackward/predictIsConsistentInType(Precision=double,Device=cpu) X Filtered by assumption.
-------------------------------------------------------------------------------------------------------------------------------------------------
nnet.checklayer.TestLayerWithoutBackward/backwardPropagationDoesNotError(Observations=one) X Filtered by assumption.
-------------------------------------------------------------------------------------------------------------------------------------------------
nnet.checklayer.TestLayerWithoutBackward/backwardPropagationDoesNotError(Observations=multiple) X Filtered by assumption.
Test Summary:
16 Passed, 2 Failed, 6 Incomplete, 10 Skipped.
Time elapsed: 1.3986 seconds.
More Answers (0)
See Also
Tags
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!An Error Occurred
Unable to complete the action because of changes made to the page. Reload the page to see its updated state.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)
Asia Pacific
- Australia (English)
- India (English)
- New Zealand (English)
- 中国
- 日本Japanese (日本語)
- 한국Korean (한국어)