Main Content

wordEmbeddingLayer

字嵌入层深度学习ing networks

expand all in page

Description

A word embedding layer maps word indices to vectors.

Use a word embedding layer in a deep learning long short-term memory (LSTM) network. An LSTM network is a type of recurrent neural network (RNN) that can learn long-term dependencies between time steps of sequence data. A word embedding layer maps a sequence of word indices to embedding vectors and learns the word embedding during training.

This layer requires Deep Learning Toolbox™.

Creation

Syntax

layer = wordEmbeddingLayer(dimension,numWords)
layer = wordEmbeddingLayer(dimension,numWords,Name,Value)

Description

example

layer= wordEmbeddingLayer(dimension,numWords)creates a word embedding layer and specifies the embedding dimension and vocabulary size.

example

layer= wordEmbeddingLayer(dimension,numWords,Name,Value)sets optionalpropertiesusing one or more name-value pairs. Enclose each property name in single quotes.

Properties

expand all

Word Embedding

Dimension of the word embedding, specified as a positive integer.

Example:300

Number of words in the model, specified as a positive integer. If the number of unique words in the training data is greater thanNumWords, then the layer maps the out-of-vocabulary words to the same vector.

Parameters and Initialization

Function to initialize the weights, specified as one of the following:

  • 'narrow-normal'– Initialize the weights by independently sampling from a normal distribution with zero mean and standard deviation 0.01.

  • 'glorot'– Initialize the weights with the Glorot initializer[1](also known as Xavier initializer). The Glorot initializer independently samples from a uniform distribution with zero mean and variance2/(numIn + numOut), wherenumIn = NumWords + 1andnumOut = Dimension.

  • 'he'– Initialize the weights with the He initializer[2]. The He initializer samples from a normal distribution with zero mean and variance2/numIn, wherenumIn = NumWords + 1.

  • 'orthogonal'– Initialize the input weights withQ, the orthogonal matrix given by the QR decomposition ofZ=QRfor a random matrixZsampled from a unit normal distribution.[3]

  • 'zeros'– Initialize the weights with zeros.

  • 'ones'– Initialize the weights with ones.

  • Function handle – Initialize the weights with a custom function. If you specify a function handle, then the function must be of the formweights = func(sz), whereszis the size of the weights.

The layer only initializes the weights when theWeightsproperty is empty.

Data Types:char|string|function_handle

Layer weights, specified as aDimension-by-NumWordsarray or aDimension-by-(NumWords+1) array.

IfWeightsis aDimension-by-NumWordsarray, then the software automatically appends an extra column for out-of-vocabulary input when training a network using thetrainNetworkfunction or when initializing adlnetworkobject.

For input integersiless than or equal toNumWords, the layer outputs the vectorWeights(:,i). Otherwise, the layer maps outputs the vectorWeights(:,NumWords+1).

Learn Rate and Regularization

Learning rate factor for the weights, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the weights in this layer. For example, ifWeightLearnRateFactoris2, then the learning rate for the weights in this layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using thetrainingOptions(Deep Learning Toolbox)function.

Data Types:single|double|int8|int16|int32|int64|uint8|uint16|uint32|uint64

L2regularization factor for the weights, specified as a nonnegative scalar.

The software multiplies this factor by the globalL2regularization factor to determine theL2regularization for the weights in this layer. For example, ifWeightL2Factoris2, then theL2regularization for the weights in this layer is twice the globalL2regularization factor. You can specify the globalL2regularization factor using thetrainingOptions(Deep Learning Toolbox)function.

Data Types:single|double|int8|int16|int32|int64|uint8|uint16|uint32|uint64

Layer

Layer name, specified as a character vector or a string scalar. ForLayerarray input, thetrainNetwork,assembleNetwork,layerGraph, anddlnetworkfunctions automatically assign names to layers withNameset to''.

Data Types:char|string

This property is read-only.

Number of inputs of the layer. This layer accepts a single input only.

Data Types:double

This property is read-only.

Input names of the layer. This layer accepts a single input only.

Data Types:cell

This property is read-only.

Number of outputs of the layer. This layer has a single output only.

Data Types:double

This property is read-only.

Output names of the layer. This layer has a single output only.

Data Types:cell

Examples

collapse all

This example uses:

Open Live Script

Create a word embedding layer with embedding dimension 300 and 5000 words.

layer = wordEmbeddingLayer(300,5000)
layer = WordEmbeddingLayer with properties: Name: '' Hyperparameters Dimension: 300 NumWords: 5000 Learnable Parameters Weights: [] Show all properties

Include a word embedding layer in an LSTM network.

inputSize = 1; embeddingDimension = 300; numWords = 5000; numHiddenUnits = 200; numClasses = 10; layers = [ sequenceInputLayer(inputSize) wordEmbeddingLayer(embeddingDimension,numWords) lstmLayer(numHiddenUnits,'OutputMode','last') fullyConnectedLayer(numClasses) softmaxLayer classificationLayer]
layers = 6x1 Layer array with layers: 1 '' Sequence Input Sequence input with 1 dimensions 2 '' Word Embedding Layer Word embedding layer with 300 dimensions and 5000 unique words 3 '' LSTM LSTM with 200 hidden units 4 '' Fully Connected 10 fully connected layer 5 '' Softmax softmax 6 '' Classification Output crossentropyex

This example uses:

Open Live Script

To initialize a word embedding layer in a deep learning network with the weights from a pretrained word embedding, use theword2vecfunction to extract the layer weights and set the'Weights'name-value pair of thewordEmbeddingLayerfunction. The word embedding layer expects columns of word vectors, so you must transpose the output of theword2vecfunction.

emb = fastTextWordEmbedding; words = emb.Vocabulary; dimension = emb.Dimension; numWords = numel(words); layer = wordEmbeddingLayer(dimension,numWords,...'Weights',word2vec(emb,words)')
layer = WordEmbeddingLayer with properties: Name: '' Hyperparameters Dimension: 300 NumWords: 999994 Learnable Parameters Weights: [300×999994 single] Show all properties

To create the corresponding word encoding from the word embedding, input the word embedding vocabulary to thewordEncodingfunction as a list of words.

enc = wordEncoding(words)
enc = wordEncoding with properties: NumWords: 999994 Vocabulary: [1×999994 string]

References

[1] Glorot, Xavier, and Yoshua Bengio. "Understanding the Difficulty of Training Deep Feedforward Neural Networks." InProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 249–356. Sardinia, Italy: AISTATS, 2010.

[2] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification." InProceedings of the 2015 IEEE International Conference on Computer Vision, 1026–1034. Washington, DC: IEEE Computer Vision Society, 2015.

[3]萨克斯,安德鲁·M。詹姆斯·l·麦克勒兰德和Surya Ganguli. "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks."arXiv preprint arXiv:1312.6120(2013).

Extended Capabilities

GPU Code Generation
Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.

See Also

(Deep Learning Toolbox)||||(Deep Learning Toolbox)|(Deep Learning Toolbox)|||

Topics

Introduced in R2018b

Text Analytics Toolbox Documentation

金宝app

Getting Started with Text Analytics in MATLAB

Getting Started with Text Analytics in MATLAB

Download now