Documentation

matlab.io.datastore.partitionable类

Package:matlab.io.datastore

向数据存储添加并行化支持金宝app

描述

matlab.io.datastore。Partitionableis an abstract mixin class that adds parallelization support to your custom datastore for use with Parallel Computing Toolbox™ andMATLAB®分布式计算服务器™

To use this mixin class, you must subclass frommatlab.io.datastore。Partitionableclass in addition to subclassing from thematlab.io.datastorebase class. Type the following syntax as the first line of your class definition file:

classdefmydatastore< matlab.io.Datastore & ... matlab.io.datastore.Partitionable ... end

To add support for parallel processing to your custom datastore, you must:

有关使用并行处理支持创建自定义数据存储的更多详细信息和步骤,请参见金宝appDevelop Custom Datastore

属性

Sealed false

要了解课堂属性,请参阅类属性

例子

expand all

Build a datastore with parallel processing support, use it to bring your custom or proprietary data into MATLAB®, and process the data in a parallel pool.

此示例使用一个简单的数据集来说明一个工作流程,您可以用它来为自己的数据构建自定义数据存储。数据集是15个二进制的集合(。bin) files where each file contains a column (1variable) and10000未签名整数的行(记录)。

dir('*。垃圾桶')
binary_data01.bin binary_data05.bin binary_data09.bin binary_data13.bin binary_data02.bin binary_data06.bin binary_data10.bin binary_data14.bin binary_data03.bin binary_data07.bin binary_data11.bin binary_data15.bin binary_data04.bin binary_data08.bin binary_data12.bin

在工作文件夹或Matlab®路径上的文件夹中实现自定义数据存储。然后创建一个新脚本,mydatastorepar.m其中包含实现自定义数据存储的代码。脚本文件的名称必须与对象构造函数函数的名称相同。例如,如果您希望构造函数函数具有MyDatastorepar的名称,则脚本文件的名称必须为mydatastorepar.m。该脚本必须包含以下步骤:

  • Step 1: Inherit from the datastore classes.

  • 步骤2:定义构造函数和所需方法。

  • 步骤3:定义您的自定义文件阅读功能。

%% STEP 1: INHERIT FROM DATASTORE CLASSESclassdefmydatastorePar < matlab.io.Datastore &。。。matlab.io.datastore。Partitionable% properties(Access = private)特性CurrentFileIndex double FileSet matlab.io.datastore.DsFileSetend%% STEP 2: DEFINE THE CONSTRUCTOR AND THE REQUIRED METHODS方法%定义您的数据存储构造函数功能myds = MyDatastorePar(location) myds.FileSet = matlab.io.datastore.DsFileSet(location,。。。“ fileextensions','.bin',。。。'FileSplitSize',8*1024);myds.CurrentFileIndex = 1;重置(myds);end% Define the hasdata method功能tf = hasdata(myds)% Return true if more data is availabletf = hasfile(myds.FileSet);end%定义读取方法功能[数据,信息] =读(myds)%读取有关提取数据的数据和信息% See also: MyFileReader()if~hasdata(myds) error(“没有更多数据”);endfileInfoTbl = nextfile(myds.FileSet); data = MyFileReader(fileInfoTbl); info.Size = size(data); info.FileName = fileInfoTbl.FileName; info.Offset = fileInfoTbl.Offset;% Update CurrentFileIndex for tracking progressiffileInfoTbl.Offset + fileInfoTbl.SplitSize >=。。。fileInfoTbl.FileSize myds.CurrentFileIndex = myds.CurrentFileIndex + 1 ;endend%定义重置方法功能reset(myds)%重置数据开始重置(myds.fileset);myds.CurrentFileIndex = 1;end%定义进度方法功能frac =进度(myds)%确定您阅读的数据百分比% from a datastorefrac = (myds.CurrentFileIndex-1)/myds.FileSet.NumFiles;end% Define the partition method功能subds = partition(myds,n,ii) subds = copy(myds); subds.FileSet = partition(myds.FileSet,n,ii); reset(subds);endend方法(访问=受保护)% If you use the FileSet property in the datastore,% then you must define the COPYELEMENT method. The% copyelement method allows methods such as readall% and preview to remain stateless功能dscopy = copyElement(ds)dscopy = copyElement@matlab.mixin.copyable(ds);dscopy.fileset = copy(ds.fileset);end% Define the maxpartitions method功能n = maxpartitions(myds) n = maxpartitions(myds.FileSet);endendend%%步骤3:实现您的自定义文件阅读功能功能data = myfilereader(fileInfotbl)% create a reader object using FileName阅读器= matlab.io.datastore.dsfileReader(fileInfotbl.fileName);%寻求偏移量搜索(读者,fileinfotbl.offset,'起源',“文件开始”);%读取fileInfotbl.splitsize数据量data = read(读取器,fileInfotbl.splitsize);end

Use your custom datastore to read data fromfolder和return the datastore object.

文件夹= fullfile ('*。垃圾桶'); ds = MyDatastorePar(folder) ;

Preview the data from the datastore.

preview(ds)
ANS = 8x1 UINT8列矢量113 180 251 91 29 66 254 214

确定数据存储的分区数。

n = numpartitions(ds);

Partition the datastore inton零件和n平行池中的工人。

parforii = 1:n subds = partition(ds,n,ii);尽管hasdata(subds) data = read(subds);% do somethingendend

Introduced in R2017b

这个话题有帮助吗?