fullyConnectedLayer
Fully connected layer
Description
A fully connected layer multiplies the input by a weight matrix and then adds a bias vector.
Creation
Description
returns a fully connected layer and specifies thelayer
= fullyConnectedLayer(outputSize
)OutputSize
property.
sets the optionalParameters and Initialization,Learning Rate and Regularization, andlayer
= fullyConnectedLayer(outputSize
,Name,Value
)Name
properties using name-value pairs. For example,fullyConnectedLayer(10,'Name','fc1')
creates a fully connected layer with an output size of 10 and the name'fc1'
。您可以指定多个名称-值对。Enclose each property name in single quotes.
Properties
Fully Connected
OutputSize
—Output size
positive integer
Output size for the fully connected layer, specified as a positive integer.
Example:10
InputSize
—Input size
'auto'
(default) |positive integer
Input size for the fully connected layer, specified as a positive integer or'auto'
。IfInputSize
is'auto'
, then the software automatically determines the input size during training.
Parameters and Initialization
WeightsInitializer
—Function to initialize weights
'glorot'
(default) |'he'
|'orthogonal'
|'narrow-normal'
|'zeros'
|'ones'
|function handle
Function to initialize the weights, specified as one of the following:
'glorot'
– Initialize the weights with the Glorot initializer[1](also known as Xavier initializer). The Glorot initializer independently samples from a uniform distribution with zero mean and variance2/(InputSize + OutputSize)
。'he'
– Initialize the weights with the He initializer[2]。The He initializer samples from a normal distribution with zero mean and variance2/InputSize
。'orthogonal'
– Initialize the input weights withQ, the orthogonal matrix given by the QR decomposition ofZ=QRfor a random matrixZsampled from a unit normal distribution.[3]'narrow-normal'
– Initialize the weights by independently sampling from a normal distribution with zero mean and standard deviation 0.01.'zeros'
– Initialize the weights with zeros.'ones'
– Initialize the weights with ones.Function handle – Initialize the weights with a custom function. If you specify a function handle, then the function must be of the form
weights = func(sz)
, wheresz
is the size of the weights. For an example, seeSpecify Custom Weight Initialization Function。
The layer only initializes the weights when theWeights
property is empty.
Data Types:char
|string
|function_handle
BiasInitializer
—Function to initialize bias
'zeros'
(default) |'narrow-normal'
|'ones'
|function handle
Function to initialize the bias, specified as one of the following:
'zeros'
— Initialize the bias with zeros.'ones'
— Initialize the bias with ones.'narrow-normal'
— Initialize the bias by independently sampling from a normal distribution with a mean of zero and a standard deviation of 0.01.Function handle — Initialize the bias with a custom function. If you specify a function handle, then the function must be of the form
bias = func(sz)
, wheresz
is the size of the bias.
The layer only initializes the bias when theBias
property is empty.
Data Types:char
|string
|function_handle
Weights
—Layer weights
[]
(default) |matrix
Layer weights, specified as a matrix.
The layer weights are learnable parameters. You can specify the initial value for the weights directly using theWeights
property of the layer. When you train a network, if theWeights
property of the layer is nonempty, thentrainNetwork
uses theWeights
property as the initial value. If theWeights
property is empty, thentrainNetwork
uses the initializer specified by theWeightsInitializer
property of the layer.
At training time,Weights
is anOutputSize
-by-InputSize
matrix.
Data Types:single
|double
Bias
—Layer biases
[]
(default) |matrix
Layer biases, specified as a matrix.
The layer biases are learnable parameters. When you train a network, ifBias
is nonempty, thentrainNetwork
uses theBias
property as the initial value. IfBias
is empty, thentrainNetwork
uses the initializer specified byBiasInitializer
。
At training time,Bias
is anOutputSize
-by-1
matrix.
Data Types:single
|double
Learning Rate and Regularization
WeightLearnRateFactor
—Learning rate factor for weights
1
(default) |nonnegative scalar
Learning rate factor for the weights, specified as a nonnegative scalar.
The software multiplies this factor by the global learning rate to determine the learning rate for the weights in this layer. For example, ifWeightLearnRateFactor
is2
, then the learning rate for the weights in this layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using thetrainingOptions
function.
Data Types:single
|double
|int8
|int16
|int32
|int64
|uint8
|uint16
|uint32
|uint64
BiasLearnRateFactor
—Learning rate factor for biases
1
(default) |nonnegative scalar
Learning rate factor for the biases, specified as a nonnegative scalar.
The software multiplies this factor by the global learning rate to determine the learning rate for the biases in this layer. For example, ifBiasLearnRateFactor
is2
, then the learning rate for the biases in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using thetrainingOptions
function.
Data Types:single
|double
|int8
|int16
|int32
|int64
|uint8
|uint16
|uint32
|uint64
WeightL2Factor
—L2regularization factor for weights
1(default) |nonnegative scalar
L2regularization factor for the weights, specified as a nonnegative scalar.
The software multiplies this factor by the globalL2regularization factor to determine theL2regularization for the weights in this layer. For example, ifWeightL2Factor
is2
, then theL2regularization for the weights in this layer is twice the globalL2regularization factor. You can specify the globalL2regularization factor using thetrainingOptions
function.
Data Types:single
|double
|int8
|int16
|int32
|int64
|uint8
|uint16
|uint32
|uint64
BiasL2Factor
—L2regularization factor for biases
0
(default) |nonnegative scalar
L2regularization factor for the biases, specified as a nonnegative scalar.
The software multiplies this factor by the globalL2regularization factor to determine theL2在这一层正规化的偏见。对example, ifBiasL2Factor
is2
, then theL2regularization for the biases in this layer is twice the globalL2regularization factor. You can specify the globalL2regularization factor using thetrainingOptions
function.
Data Types:single
|double
|int8
|int16
|int32
|int64
|uint8
|uint16
|uint32
|uint64
Layer
Name
—Layer name
''
(default) |character vector|string scalar
Layer name, specified as a character vector or a string scalar. ForLayer
array input, thetrainNetwork
,assembleNetwork
,layerGraph
, anddlnetwork
functions automatically assign names to layers with name''
。
Data Types:char
|string
NumInputs
—Number of inputs
1
(default)
This property is read-only.
Number of inputs of the layer. This layer accepts a single input only.
Data Types:double
InputNames
—Input names
{'in'}
(default)
This property is read-only.
Input names of the layer. This layer accepts a single input only.
Data Types:cell
NumOutputs
—Number of outputs
1
(default)
This property is read-only.
Number of outputs of the layer. This layer has a single output only.
Data Types:double
OutputNames
—Output names
{'out'}
(default)
This property is read-only.
Output names of the layer. This layer has a single output only.
Data Types:cell
Examples
Create Fully Connected Layer
Create a fully connected layer with an output size of 10 and the name'fc1'
。
layer = fullyConnectedLayer(10,'Name','fc1')
layer = FullyConnectedLayer with properties: Name: 'fc1' Hyperparameters InputSize: 'auto' OutputSize: 10 Learnable Parameters Weights: [] Bias: [] Show all properties
Include a fully connected layer in aLayer
array.
layers = [。..imageInputLayer([28 28 1]) convolution2dLayer(5,20) reluLayer maxPooling2dLayer(2,'Stride',2) fullyConnectedLayer(10) softmaxLayer classificationLayer]
layers = 7x1 Layer array with layers: 1 '' Image Input 28x28x1 images with 'zerocenter' normalization 2 '' Convolution 20 5x5 convolutions with stride [1 1] and padding [0 0 0 0] 3 '' ReLU ReLU 4 '' Max Pooling 2x2 max pooling with stride [2 2] and padding [0 0 0 0] 5 '' Fully Connected 10 fully connected layer 6 '' Softmax softmax 7 '' Classification Output crossentropyex
Specify Initial Weights and Biases in Fully Connected Layer
To specify the weights and bias initializer functions, use theWeightsInitializer
andBiasInitializer
properties respectively. To specify the weights and biases directly, use theWeights
andBias
properties respectively.
Specify Initialization Function
Create a fully connected layer with an output size of 10 and specify the weights initializer to be the He initializer.
outputSize = 10; layer = fullyConnectedLayer(outputSize,'WeightsInitializer','he')
layer = FullyConnectedLayer with properties: Name: '' Hyperparameters InputSize: 'auto' OutputSize: 10 Learnable Parameters Weights: [] Bias: [] Show all properties
Note that theWeights
andBias
properties are empty. At training time, the software initializes these properties using the specified initialization functions.
Specify Custom Initialization Function
To specify your own initialization function for the weights and biases, set theWeightsInitializer
andBiasInitializer
properties to a function handle. For these properties, specify function handles that take the size of the weights and biases as input and output the initialized value.
Create a fully connected layer with output size 10 and specify initializers that sample the weights and biases from a Gaussian distribution with a standard deviation of 0.0001.
outputSize = 10; weightsInitializationFcn = @(sz) rand(sz) * 0.0001; biasInitializationFcn = @(sz) rand(sz) * 0.0001; layer = fullyConnectedLayer(outputSize,。..'WeightsInitializer',@(sz) rand(sz) * 0.0001,。..'BiasInitializer',@(sz) rand(sz) * 0.0001)
layer = FullyConnectedLayer with properties: Name: '' Hyperparameters InputSize: 'auto' OutputSize: 10 Learnable Parameters Weights: [] Bias: [] Show all properties
Again, theWeights
andBias
properties are empty. At training time, the software initializes these properties using the specified initialization functions.
Specify Weights and Bias Directly
Create a fully connected layer with an output size of 10 and set the weights and bias toW
andb
in the MAT fileFCWeights.mat
respectively.
outputSize = 10; loadFCWeightslayer = fullyConnectedLayer(outputSize,。..'Weights',W,。..'Bias',b)
layer = FullyConnectedLayer with properties: Name: '' Hyperparameters InputSize: 720 OutputSize: 10 Learnable Parameters Weights: [10x720 double] Bias: [10x1 double] Show all properties
Here, theWeights
andBias
properties contain the specified values. At training time, if these properties are non-empty, then the software uses the specified values as the initial weights and biases. In this case, the software does not use the initializer functions.
Algorithms
Fully Connected Layer
A fully connected layer multiplies the input by a weight matrix and then adds a bias vector.
The convolutional (and down-sampling) layers are followed by one or more fully connected layers.
As the name suggests, all neurons in a fully connected layer connect to all the neurons in the previous layer. This layer combines all of the features (local information) learned by the previous layers across the image to identify the larger patterns. For classification problems, the last fully connected layer combines the features to classify the images. This is the reason that theoutputSize
argument of the last fully connected layer of the network is equal to the number of classes of the data set. For regression problems, the output size must be equal to the number of response variables.
You can also adjust the learning rate and the regularization parameters for this layer using the related name-value pair arguments when creating the fully connected layer. If you choose not to adjust them, thentrainNetwork
uses the global training parameters defined by thetrainingOptions
function. For details on global and layer training options, seeSet Up Parameters and Train Convolutional Neural Network。
A fully connected layer multiplies the input by a weight matrixWand then adds a bias vectorb。
如果the input to the layer is a sequence (for example, in an LSTM network), then the fully connected layer acts independently on each time step. For example, if the layer before the fully connected layer outputs an arrayXof sizeD-by-N-by-S, then the fully connected layer outputs an arrayZof sizeoutputSize
-by-N-by-S。At time stept, the corresponding entry ofZis
, where
denotes time steptofX。
The fully connected layer flattens the output. It reshapes the array such that the spatial data is encoded in the channel dimension.
For sequence input, the layer applies the fully connect operation independently to each time step of the input.
Layer Input and Output Formats
Layers in a layer array or layer graph pass data specified as formatteddlarray
objects.
You can interact with thesedlarray
objects in automatic differentiation workflows such as when developing a custom layer, using afunctionLayer
object, or using theforward
andpredict
functions withdlnetwork
objects.
This table shows the supported input formats ofFullyConnectedLayer
对象和相应的输出格式。如果the output of the layer is passed to a custom layer that does not inherit from thennet.layer.Formattable
class, or aFunctionLayer
object with theFormattable
option set tofalse
, then the layer receives an unformatteddlarray
object with dimensions ordered corresponding to the formats outlined in this table.
Input Format | Output Format |
---|---|
|
|
|
|
|
|
|
|
|
|
Indlnetwork
objects,FullyConnectedLayer
objects also support the following input and output format combinations.
Input Format | Output Format |
---|---|
|
|
|
|
|
|
To use these input formats intrainNetwork
workflows, first convert the data to"CBT"
(channel, batch, time) format usingflattenLayer
。
References
[1] Glorot, Xavier, and Yoshua Bengio. "Understanding the Difficulty of Training Deep Feedforward Neural Networks." InProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 249–356. Sardinia, Italy: AISTATS, 2010.
[2] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification." InProceedings of the 2015 IEEE International Conference on Computer Vision, 1026–1034. Washington, DC: IEEE Computer Vision Society, 2015.
[3]萨克斯,安德鲁·M。詹姆斯L. McClelland, and Surya Ganguli. "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks."arXiv preprint arXiv:1312.6120(2013).
Extended Capabilities
C/C++ Code Generation
Generate C and C++ code using MATLAB® Coder™.
GPU Code Generation
Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.
Version History
Introduced in R2016aR2019a: Default weights initialization is Glorot
Behavior changed in R2019a
Starting in R2019a, the software, by default, initializes the layer weights of this layer using the Glorot initializer. This behavior helps stabilize training and usually reduces the training time of deep networks.
In previous releases, the software, by default, initializes the layer weights by sampling from a normal distribution with zero mean and variance 0.01. To reproduce this behavior, set the'WeightsInitializer'
option of the layer to'narrow-normal'
。
Abrir ejemplo
Tiene una versión modificada de este ejemplo. ¿Desea abrir este ejemplo con sus modificaciones?
Comando de MATLAB
Ha hecho clic en un enlace que corresponde a este comando de MATLAB:
Ejecute el comando introduciéndolo en la ventana de comandos de MATLAB. Los navegadores web no admiten comandos de MATLAB.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select:。
You can also select a web site from the following list:
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina(Español)
- Canada(English)
- United States(English)
Europe
- Belgium(English)
- Denmark(English)
- Deutschland(Deutsch)
- España(Español)
- Finland(English)
- France(Français)
- Ireland(English)
- Italia(Italiano)
- Luxembourg(English)
- Netherlands(English)
- Norway(English)
- Österreich(Deutsch)
- Portugal(English)
- Sweden(English)
- Switzerland
- United Kingdom(English)