Define Custom Deep Learning Layers

Tip

This topic explains how to define custom deep learning layers for your problems. For a list of built-in layers in Deep Learning Toolbox™, seeList of Deep Learning Layers.

You can define your own custom deep learning layers for your task. You can specify a custom loss function using a custom output layer and define custom layers with or without learnable and state parameters. After defining a custom layer, you can check that the layer is valid and GPU compatible, and outputs correctly defined gradients.

This topic explains the architecture of deep learning layers and how to define custom layers to use for your tasks.

Type	Description
Intermediate layer	Define a custom deep learning layer and specify optional learnable parameters and state parameters. For more information, seeDefine Custom Deep Learning Intermediate Layers. For an example showing how to define a custom layer with learnable parameters, seeDefine Custom Deep Learning Layer with Learnable Parameters. For an example showing how to define a custom layer with multiple inputs, seeDefine Custom Deep Learning Layer with Multiple Inputs.
Classification output layer	Define a custom classification output layer and specify a loss function. For more information, seeDefine Custom Deep Learning Output Layers. For an example showing how to define a custom classification output layer and specify a loss function, seeDefine Custom Classification Output Layer.
Regression output layer	Define a custom regression output layer and specify a loss function. For more information, seeDefine Custom Deep Learning Output Layers. For an example showing how to define a custom regression output layer and specify a loss function, seeDefine Custom Regression Output Layer.

Type

Description

Intermediate layer

Define a custom deep learning layer and specify optional learnable parameters and state parameters.

For more information, seeDefine Custom Deep Learning Intermediate Layers.

For an example showing how to define a custom layer with learnable parameters, seeDefine Custom Deep Learning Layer with Learnable Parameters. For an example showing how to define a custom layer with multiple inputs, seeDefine Custom Deep Learning Layer with Multiple Inputs.

Classification output layer

Define a custom classification output layer and specify a loss function.

For more information, seeDefine Custom Deep Learning Output Layers.

For an example showing how to define a custom classification output layer and specify a loss function, seeDefine Custom Classification Output Layer.

Regression output layer

Define a custom regression output layer and specify a loss function.

For more information, seeDefine Custom Deep Learning Output Layers.

For an example showing how to define a custom regression output layer and specify a loss function, seeDefine Custom Regression Output Layer.

Layer Templates

You can use the following templates to define new layers.

Intermediate Layer Template

This template outlines the structure of an intermediate layer. For more information, seeDefine Custom Deep Learning Intermediate Layers.

For an example showing how to define a layer with learnable parameters, seeDefine Custom Deep Learning Layer with Learnable Parameters.

classdefmyLayer < nnet.layer.Layer% ...% & nnet.layer.Formattable ... % (Optional)% & nnet.layer.Acceleratable % (Optional)properties% (Optional) Layer properties.% Declare layer properties here.结束properties(可学的)% (Optional) Layer learnable parameters.% Declare learnable parameters here.结束properties(State)% (Optional) Layer state parameters.% Declare state parameters here.结束properties(可学的, State)% (Optional) Nested dlnetwork objects with both learnable% parameters and state parameters.% Declare nested networks with learnable and state parameters here.结束methodsfunctionlayer = myLayer()% (Optional) Create a myLayer.% This function must have the same name as the class.% Define layer constructor function here.结束function[Z,state] = predict(layer,X)% Forward input data through the layer at prediction time and% output the result and updated state.%% Inputs:% layer - Layer to forward propagate through% X - Input data% Outputs:% Z - Output of layer forward function% state - (Optional) Updated layer state%% - For layers with multiple inputs, replace X with X1,...,XN,% where N is the number of inputs.% - For layers with multiple outputs, replace Z with% Z1,...,ZM, where M is the number of outputs.% - For layers with multiple state parameters, replace state% with state1,...,stateK, where K is the number of state% parameters.% Define layer predict function here.结束function[Z,state,memory] = forward(layer,X)% (Optional) Forward input data through the layer at training%的时间和输出结果,更新state, and a memory%值。%% Inputs:% layer - Layer to forward propagate through% X - Layer input data% Outputs:% Z - Output of layer forward function% state - (Optional) Updated layer state% memory - (Optional) Memory value for custom backward% function%% - For layers with multiple inputs, replace X with X1,...,XN,% where N is the number of inputs.% - For layers with multiple outputs, replace Z with% Z1,...,ZM, where M is the number of outputs.% - For layers with multiple state parameters, replace state% with state1,...,stateK, where K is the number of state% parameters.% Define layer forward function here.结束functionlayer = resetState(layer)% (Optional) Reset layer state.% Define reset state function here.结束function[dLdX,dLdW,dLdSin] = backward(layer,X,Z,dLdZ,dLdSout,memory)% (Optional) Backward propagate the derivative of the loss% function through the layer.%% Inputs:% layer - Layer to backward propagate through% X - Layer input data% Z - Layer output data% dLdZ - Derivative of loss with respect to layer% output% dLdSout - (Optional) Derivative of loss with respect% to state output% memory - Memory value from forward function% Outputs:% dLdX - Derivative of loss with respect to layer input% dLdW - (Optional) Derivative of loss with respect to% learnable parameter% dLdSin - (Optional) Derivative of loss with respect to% state input%% - For layers with state parameters, the backward syntax must% include both dLdSout and dLdSin, or neither.% - For layers with multiple inputs, replace X and dLdX with% X1,...,XN and dLdX1,...,dLdXN, respectively, where N is% the number of inputs.% - For layers with multiple outputs, replace Z and dlZ with% Z1,...,ZM and dLdZ,...,dLdZM, respectively, where M is the% number of outputs.% - For layers with multiple learnable parameters, replace% dLdW with dLdW1,...,dLdWP, where P is the number of% learnable parameters.% - For layers with multiple state parameters, replace dLdSin% and dLdSout with dLdSin1,...,dLdSinK and% dLdSout1,...,dldSoutK, respectively, where K is the number% of state parameters.% Define layer backward function here.结束结束结束

Classification Output Layer Template

Regression Output Layer Template

Intermediate Layer Architecture

During training, the software iteratively performs forward and backward passes through the network.

During a forward pass through the network, each layer takes the outputs of the previous layers, applies a function, and then outputs (forward propagates) the results to the next layers. Stateful layers, such as LSTM layers, also update the layer state.

Layers can have multiple inputs or outputs. For example, a layer can takeX₁, …,X_Nfrom multiple previous layers and forward propagate the outputsZ₁, …,Z_Mto subsequent layers.

At the end of a forward pass of the network, the output layer calculates the lossLbetween the predictionsYand the targetsT.

During the backward pass through the network, each layer takes the derivatives of the loss with respect to the outputs of the layer, computes the derivatives of the lossLwith respect to the inputs, and then backward propagates the results. If the layer has learnable parameters, then the layer also computes the derivatives of the layer weights (learnable parameters). The layer uses the derivatives of the weights to update the learnable parameters.

The following figure describes the flow of data through a deep neural network and highlights the data flow through a layer with a single inputX, a single outputZ, and a learnable parameterW.

For more information about custom intermediate layers, seeDefine Custom Deep Learning Intermediate Layers.

Output Layer Architecture

At the end of a forward pass at training time, an output layer takes the outputsYof the previous layer (the network predictions) and calculates the lossLbetween these predictions and the training targets. The output layer computes the derivatives of the lossLwith respect to the predictionsYand outputs (backward propagates) results to the previous layer.

The following figure describes the flow of data through a neural network and an output layer.

For more information, seeDefine Custom Deep Learning Output Layers.

Check Validity of Custom Layer

If you create a custom deep learning layer, then you can use thecheckLayerfunction to check that the layer is valid. The function checks layers for validity, GPU compatibility, correctly defined gradients, and code generation compatibility. To check that a layer is valid, run the following command:

checkLayer(layer,validInputSize)

layeris an instance of the layer andvalidInputSizeis a vector or cell array specifying the valid input sizes to the layer. To check with multiple observations, use theObservationDimensionoption. To run the check for code generation compatibility, set theCheckCodegenCompatibilityoption to1(true).For large input sizes, the gradient checks take longer to run. To speed up the check, specify a smaller valid input size.

For more information, seeCheck Custom Layer Validity.