Load the factory reports data and create atokenizedDocumentarray.

文件名="factoryReports.csv"; data = readtable(filename,'TextType','string'); textData = data.Description; documents = tokenizedDocument(textData);

Create a word encoding.

enc = wordEncoding(documents);

Convert the documents to sequences of word indices.

sequences = doc2sequence(enc,documents);

View the sizes of the first 10 sequences. Each sequence is a 1-by-Svector, whereSis the number of word indices in the sequence. Because the sequences are padded,Sis constant.

sequences(1:10)

ans=10×1 cell array{1x17 double} {1x17 double} {1x17 double} {1x17 double} {1x17 double} {1x17 double} {1x17 double} {1x17 double} {1x17 double} {1x17 double}

Convert an array of tokenized documents to sequences of word vectors using a pretrained word embedding.

Load a pretrained word embedding using thefastTextWordEmbeddingfunction. This function requires Text Analytics Toolbox™ Modelfor fastText English 16 Billion Token Word Embeddingsupport package. If this support package is not installed, then the function provides a download link.

emb = fastTextWordEmbedding;

Load the factory reports data and create atokenizedDocumentarray.

文件名="factoryReports.csv"; data = readtable(filename,'TextType','string'); textData = data.Description; documents = tokenizedDocument(textData);

Convert the documents to sequences of word vectors usingdoc2sequence。Thedoc2sequencefunction, by default, left-pads the sequences to have the same length. When converting large collections of documents using a high-dimensional word embedding, padding can require large amounts of memory. To prevent the function from padding the data, set the'PaddingDirection'option to'none'。Alternatively, you can control the amount of padding using the'Length'option.

sequences = doc2sequence(emb,documents,'PaddingDirection','none');

View the sizes of the first 10 sequences. Each sequence isD-by-Smatrix, whereDis the embedding dimension, andSis the number of word vectors in the sequence.

sequences(1:10)

ans=10×1 cell array{300×10 single} {300×11 single} {300×11 single} {300×6 single} {300×5 single} {300×10 single} {300×8 single} {300×9 single} {300×7 single} {300×13 single}

Convert a collection of documents to sequences of word vectors using a pretrained word embedding, and pad or truncate the sequences to a specified length.

Load a pretrained word embedding usingfastTextWordEmbedding。This function requires Text Analytics Toolbox™ Modelfor fastText English 16 Billion Token Word Embeddingsupport package. If this support package is not installed, then the function provides a download link.

emb = fastTextWordEmbedding;

Load the factory reports data and create atokenizedDocumentarray.

文件名="factoryReports.csv"; data = readtable(filename,'TextType','string'); textData = data.Description; documents = tokenizedDocument(textData);

Convert the documents to sequences of word vectors. Specify to left-pad or truncate the sequences to have length 100.

sequences = doc2sequence(emb,documents,'Length',100);

View the sizes of the first 10 sequences. Each sequence isD-by-Smatrix, whereDis the embedding dimension, andSis the number of word vectors in the sequence (the sequence length). Because the sequence length is specified,Sis constant.

sequences(1:10)

ans=10×1 cell array{300×100 single} {300×100 single} {300×100 single} {300×100 single} {300×100 single} {300×100 single} {300×100 single} {300×100 single} {300×100 single} {300×100 single}