What Is a Datastore?

数据存储中读取一个文件是一个对象or a collection of files or data. The datastore acts as a repository for data that has the same structure and formatting. For example, each file in a datastore must contain data of the same type (such as numeric or text) appearing in the same order, and separated by the same delimiter.

A datastore is useful when:

You can create a datastore based on the type of data or application. The different types of datastores contain properties pertinent to the type of data that they support. For example, see the following table for a list of MATLAB^®datastores. For a complete list of datastores, seeSelect Datastore for File Format or Application.

Create and Read from a Datastore

Use thetabularTextDatastorefunction to create a datastore from the sample fileairlinesmall.csv, which contains departure and arrival information about individual airline flights. The result is aTabularTextDatastoreobject.

ds = tabularTextDatastore('airlinesmall.csv')

ds = TabularTextDatastore with properties: Files: { ' ...\matlab\toolbox\matlab\demos\airlinesmall.csv' } Folders: { ' ...\matlab\toolbox\matlab\demos' } FileEncoding: 'UTF-8' AlternateFileSystemRoots: {} PreserveVariableNames: false ReadVariableNames: true VariableNames: {'Year', 'Month', 'DayofMonth' ... and 26 more} DatetimeLocale: en_US Text Format Properties: NumHeaderLines: 0 Delimiter: ',' RowDelimiter: '\r\n' TreatAsMissing: '' MissingValue: NaN Advanced Text Format Properties: TextscanFormats: {'%f', '%f', '%f' ... and 26 more} TextType: 'char' ExponentCharacters: 'eEdD' CommentStyle: '' Whitespace: ' \b\t' MultipleDelimitersAsOne: false Properties that control the table returned by preview, read, readall: SelectedVariableNames: {'Year', 'Month', 'DayofMonth' ... and 26 more} SelectedFormats: {'%f', '%f', '%f' ... and 26 more} ReadSize: 20000 rows OutputType: 'table' RowTimes: [] Write-specific Properties: SupportedOutputFormats: ["txt" "csv" "xlsx" "xls" "parquet" "parq"] DefaultOutputFormat: "txt"

After creating the datastore, you can preview the data without having to load it all into memory. You can specify variables (columns) of interest using theSelectedVariableNamesproperty to preview or read only those variables.

ds.SelectedVariableNames = {'DepTime','DepDelay'}; preview(ds)

ans = 8×2 table DepTime DepDelay _______ ________ 642 12 1021 1 2055 20 1332 12 629 -1 1446 63 928 -2 859 -1

You can specify the values in your data which represent missing values. Inairlinesmall.csv, missing values are represented byNA.

ds.TreatAsMissing ='NA';

If all of the data in the datastore for the variables of interest fit in memory, you can read it using thereadallfunction.

T = readall(ds);

Otherwise, read the data in smaller subsets that do fit in memory, using thereadfunction. By default, thereadfunction reads from aTabularTextDatastore20,000 rows at a time. However, you can change this value by assigning a new value to theReadSizeproperty.

ds.ReadSize = 15000;

Reset the datastore to the initial state before re-reading, using theresetfunction. By calling thereadfunction within awhileloop, you can perform intermediate calculations on each subset of data, and then aggregate the intermediate results at the end. This code calculates the maximum value of theDepDelayvariable.

reset(ds) X = [];whilehasdata(ds) T = read(ds); X(end+1) = max(T.DepDelay);endmaxDelay = max(X)

maxDelay = 1438

If the data in each individual file fits in memory, you can specify that each call toreadshould read one complete file rather than a specific number of rows.

reset(ds) ds.ReadSize ='file'; X = [];whilehasdata(ds) T = read(ds); X(end+1) = max(T.DepDelay);endmaxDelay = max(X);

In addition to reading subsets of data in a datastore, you can apply map and reduce functions to the datastore usingmapreduceor create a tall array usingtall. For more information, seeGetting Started with MapReduceandTall Arrays for Out-of-Memory Data.