Add token type details to documents
detects the token types inupdatedDocuments
= addTypeDetails(documents
)documents
and updates the token details. The function adds type details to the tokens with unknown type only. To get the token types fromupdatedDocuments
, usetokenDetails
.
specifies additional options using one or more name-value pairs.updatedDocuments
= addTypeDetails(documents
,Name,Value
)
Tip
UseaddTypeDetails
before using thelower
,upper
, anderasePunctuation
functions asaddTypeDetails
uses information that is removed by these functions.
tokenizedDocument
|tokenDetails
|addSentenceDetails
|addPartOfSpeechDetails
|splitSentences
|abbreviations
|topLevelDomains
|corpusLanguage
|addLanguageDetails
|addLemmaDetails
|addEntityDetails