geluLayer

Gaussian error linear unit (GELU) layer

Since R2022b

Description

A Gaussian error linear unit (GELU) layer weights the input by its probability under a Gaussian distribution.

This operation is given by

$$\text{GELU}(x)=\frac{x}{2}\left(1+\text{}\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right),$$

where erf denotes the error function.

Creation

Syntax

layer = geluLayer

layer = geluLayer(Name=Value)

Description

layer = geluLayer returns a GELU layer.

layer = geluLayer(Name=Value) sets the optional Approximation and Name properties using name-value arguments. For example, geluLayer(Name="gelu") creates a GELU layer with the name "gelu".

Properties

GELU

Approximation — Approximation method for GELU operation

'none' (default) | 'tanh'

Approximation method for the GELU operation, specified as one of these values:

• 'none' — Do not use approximation.

• 'tanh' — Approximate the underlying error function using

  $$\text{erf}\left(\frac{x}{\sqrt{2}}\right)\approx \text{tanh}\left(\sqrt{\frac{2}{\pi }}\left(x+0.044715x^3\right)\right).$$

Tip

In MATLAB^®, computing the tanh approximation is typically less accurate, and, for large input sizes, slower than computing the GELU activation without using an approximation. Use the tanh approximation when you want to reproduce models that use this approximation, such as BERT and GPT-2.

Layer

Name — Layer name

'' (default) | character vector | string scalar

Layer name, specified as a character vector or a string scalar. For Layer array input, the trainNetwork, assembleNetwork, layerGraph, and dlnetwork functions automatically assign names to layers with the name ''.

Data Types: char | string

NumInputs — Number of inputs

1 (default)

This property is read-only.

Number of inputs of the layer. This layer accepts a single input only.

Data Types: double

InputNames — Input names

{"in"} (default)

This property is read-only.

Input names of the layer. This layer accepts a single input only.

Data Types: cell

NumOutputs — Number of outputs

1 (default)

This property is read-only.

Number of outputs of the layer. This layer has a single output only.

Data Types: double

OutputNames — Output names

{'out'} (default)

This property is read-only.

Output names of the layer. This layer has a single output only.

Data Types: cell

Examples

Create GELU Layer

Create a GELU layer.

layer = geluLayer

layer = 
  GELULayer with properties:

             Name: ''

   Hyperparameters
    Approximation: 'none'

Include a GELU layer in a Layer array.

layers = [
    imageInputLayer([28 28 1])
    convolution2dLayer(5,20)
    geluLayer
    maxPooling2dLayer(2,Stride=2)
    fullyConnectedLayer(10)
    softmaxLayer
    classificationLayer]

layers = 
  7×1 Layer array with layers:

     1   ''   Image Input             28×28×1 images with 'zerocenter' normalization
     2   ''   Convolution             20 5×5 convolutions with stride [1  1] and padding [0  0  0  0]
     3   ''   GELU                    GELU
     4   ''   Max Pooling             2×2 max pooling with stride [2  2] and padding [0  0  0  0]
     5   ''   Fully Connected         10 fully connected layer
     6   ''   Softmax                 softmax
     7   ''   Classification Output   crossentropyex

Algorithms

Gaussian Error Linear Unit Activation

The Gaussian error linear unit (GELU) activation operation weights the input by its probability under a Gaussian distribution.

This operation is given by

$$\text{GELU}(x)=\frac{x}{2}\left(1+\text{}\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right),$$

where erf denotes the error function given by

$$\text{erf}(x)=\frac{2}{\sqrt{\pi }}{\displaystyle {\int }_0^x{e}^{-t^2}dt}.$$

When the Approximation option is "tanh", the software approximates the error function using

$$\text{erf}\left(\frac{x}{\sqrt{2}}\right)\approx \text{tanh}\left(\sqrt{\frac{2}{\pi }}\left(x+0.044715x^3\right)\right).$$

Layer Input and Output Formats

Layers in a layer array or layer graph pass data to subsequent layers as formatted dlarray objects. The format of a dlarray object is a string of characters, in which each character describes the corresponding dimension of the data. The formats consists of one or more of these characters:

• "S" — Spatial

• "C" — Channel

• "B" — Batch

• "T" — Time

• "U" — Unspecified

For example, 2-D image data represented as a 4-D array, where the first two dimensions correspond to the spatial dimensions of the images, the third dimension corresponds to the channels of the images, and the fourth dimension corresponds to the batch dimension, can be described as having the format "SSCB" (spatial, spatial, channel, batch).

GELULayer objects apply an element-wise operation and supports input data of any format. The layer and does not add or remove any dimensions, so it outputs data with the same format as its inputs.

References

[1] Hendrycks, Dan, and Kevin Gimpel. "Gaussian error linear units (GELUs)." Preprint, submitted June 27, 2016. https://arxiv.org/abs/1606.08415

Version History

Introduced in R2022b