手势识别使用视频和深度学习

downloadFolder = fullfile (tempdir,“姿态”);如果~ isfolder (downloadFolder) mkdir (downloadFolder);结束zipFile =“slowFastPretrained_fourClasses.zip”;如果~ isfile (fullfile (downloadFolder zipFile) disp (“下载pretrained网络…”);downloadURL =“https://ssd.mathworks.com/金宝appsupportfiles/vision/data/”+ zipFile;zipFile = fullfile (downloadFolder zipFile);websave (zipFile downloadURL);解压缩(zipFile downloadFolder);disp (“下载”。)结束

pretrainedDataFile = fullfile (downloadFolder,“slowFastPretrained_fourClasses.mat”);pretrained =负载(pretrainedDataFile);slowFastClassifier = pretrained.data.slowFast;

类= slowFastClassifier.Classes

videoFilename = fullfile (downloadFolder,“waving-hello.avi”);videoReader = videoReader (videoFilename);放像机= vision.VideoPlayer;放像机。Name =“打招呼”;而hasFrame (videoReader)帧= readFrame (videoReader);步骤(放像机、框架);结束释放(放像机);

numSequences = 10;

[gestureLabel,分数]= classifyVideoFile (slowFastClassifier、videoFilename NumSequences = NumSequences)

doTraining = false;

groundTruthFolder = fullfile (downloadFolder,“groundTruthFolder”);如果~ isfolder (groundTruthFolder) mkdir (groundTruthFolder);结束

zipFile =“videoClipsAndSceneLabels.zip”;如果~ isfile (fullfile (groundTruthFolder zipFile) disp (“地面实况训练数据的下载…”);downloadURL =“https://ssd.mathworks.com/金宝appsupportfiles/vision/data/”+ zipFile;zipFile = fullfile (groundTruthFolder zipFile);websave (zipFile downloadURL);解压缩(zipFile groundTruthFolder);结束

groundTruthFolder = fullfile (downloadFolder,“groundTruthFolder”);trainingFolder = fullfile (downloadFolder,“videoScenes”);extractVideoScenes (groundTruthFolder trainingFolder类);

numFrames = 16;

frameSize = [112112];

numChannels = 3;

isDataForTraining = true;dsTrain = createFileDatastore (trainingFolder numFrames numChannels,类,isDataForTraining);

baseNetwork =“resnet50-3d”;

inputSize = [frameSize、numChannels numFrames];

slowFast = slowFastVideoClassifier (baseNetwork字符串(类),InputSize = InputSize);

slowFast。ModelName =“使用深度学习手势识别器”;

dsTrain =变换(dsTrain @augmentVideo);

preprocessInfo。统计= slowFast.InputNormalizationStatistics;preprocessInfo。InputSize=我nputSize; preprocessInfo.SizingOption =“调整”;dsTrain =变换(dsTrain @(数据)preprocessVideoClips(数据、preprocessInfo));

参数个数。类=类;参数个数。MiniBatchSize = 5;参数个数。NumIterations = 600;参数个数。CosineNumIterations = (100 200 300);参数个数。SaveBestAfterIteration = 400; params.MinLearningRate = 1e-4; params.MaxLearningRate = 1e-3; params.Momentum = 0.9; params.Velocity = []; params.L2Regularization = 0.0005; params.ProgressPlot = false; params.Verbose = true; params.DispatchInBackground = true; params.NumWorkers = 12;

参数个数。ModelFilename =“slowFastPretrained_fourClasses.mat”;如果doTraining时代= 1;bestLoss =最大浮点数;accTrain = [];lossTrain = [];迭代= 1;开始=抽搐;火车离站时刻表=开始;打乱= shuffleTrainDs (dsTrain);%的输出是两个:一个用于RGB框架,和一个地面真理标签。numOutputs = 2;兆贝可= createMiniBatchQueue(重组、numOutputs params);%使用initializeTrainingProgressPlot和initializeVerboseOutput%支金宝app持功能,列出的例子,来初始化%的培训发展情节和详细输出显示培训%,训练精度,验证精度。策划者= initializeTrainingProgressPlot (params);initializeVerboseOutput (params);而迭代< = params.NumIterations%遍历的数据集。[dlX1,海底]=下一个(兆贝可);%计算模型使用dlfeval梯度和损失。(渐变、失去、acc) =…dlfeval (@modelGradients, slowFast dlX1,海底);%累积损失和精度。lossTrain = (lossTrain、损失);accTrain = (accTrain, acc);%更新网络状态。slowFast。=状态;%更新梯度和视频分类器的参数%使用个优化器。[slowFast,参数个数。速度,learnRate] =…updateLearnables (slowFast、渐变参数、params.Velocity迭代);如果~ hasdata(兆贝可)| | = = params.NumIterations迭代%完成当前的时代。做验证和更新进展。火车离站时刻表= toc(火车离站时刻表);accTrain =意味着(accTrain);lossTrain =意味着(lossTrain);%更新培训的进展。displayVerboseOutputEveryEpoch(参数、启动、learnRate时代,迭代,…accTrain lossTrain,火车离站时刻表);updateProgressPlot (params,策划者,时代,迭代,开始,lossTrain, accTrain);%保存训练视频分类器参数,这给了%到目前为止最好的训练损失。使用saveData支持函数,金宝app%的结尾部分,本文罗列了这个例子。bestLoss = saveData (slowFast bestLoss,迭代,lossTrain params);结束如果~ hasdata(兆贝可)& & < params.NumIterations迭代%完成当前的时代。初始化培训损失,准确性%值,minibatchqueue为下一个时代。accTrain = [];lossTrain = [];时代=时代+ 1;火车离站时刻表=抽搐;打乱= shuffleTrainDs (dsTrain);兆贝可= createMiniBatchQueue(重组、numOutputs params);结束迭代=迭代+ 1;结束%完成训练时显示一条消息。endVerboseOutput (params);disp (”模型保存到:“+ params.ModelFilename);结束

isDataForTraining = false;dsEval = createFileDatastore (trainingFolder numFrames numChannels,类,isDataForTraining);dsEval =变换(dsEval @(数据)preprocessVideoClips(数据、preprocessInfo));

如果doTraining transferLearned =负载(params.ModelFilename);slowFastClassifier = transferLearned.data.slowFast;结束

numOutputs = 2;兆贝可= createMiniBatchQueue (dsEval、numOutputs params);

numClasses =元素个数(params.Classes);cmat =稀疏(numClasses numClasses);而hasdata(兆贝可)[dlVideo,海底]=(兆贝可);%的计算机预测SlowFast训练%视频分类器。dlYPred =预测(slowFastClassifier dlVideo);dlYPred = squeezeIfNeeded (dlYPred、海底);通过使用最大%总混淆矩阵%值预测分数和地面真理的标签。[~,欧美]= max(海底,[],1);[~,YPred] = max (dlYPred [], 1);欧美,cmat = aggregateConfusionMetric (cmat YPred);结束

evalClipAccuracy =总和(诊断接头(cmat)。/笔(cmat,“所有”)

图图= confusionchart (cmat、类);

函数数据存储= createFileDatastore (trainingFolder numFrames numChannels,类,isDataForTraining) readFcn = @ (f, u) readVideo (f, u, numFrames numChannels,类,isDataForTraining);数据存储= fileDatastore (trainingFolder,…“IncludeSubfolders”,真的,…“FileExtensions”,“.avi”,…“ReadFcn”readFcn,…“ReadMode”,“partialfile”);结束

函数打乱= shuffleTrainDs (dsTrain)打乱= (dsTrain)复印件;改变= isa(打乱,“matlab.io.datastore.TransformedDatastore”);如果转换文件= shuffled.UnderlyingDatastores {1} .Files;其他的文件= shuffled.Files;结束n =元素个数(文件);shuffledIndices = randperm (n);如果改变shuffled.UnderlyingDatastores {1}。文件=文件(shuffledIndices);其他的重新洗了一遍。F我le年代=f我le年代(年代huffledIndices);结束重置(重组);结束

函数(数据、用户数据做)= readVideo(文件名,用户数据、numFrames numChannels,类,isDataForTraining)如果isempty(用户数据)用户数据。re一个der=VideoReader(f我len一个me); userdata.batchesRead = 0; userdata.label = getLabel(filename,classes); totalFrames = floor(userdata.reader.Duration * userdata.reader.FrameRate); totalFrames = min(totalFrames, userdata.reader.NumFrames); userdata.totalFrames = totalFrames; userdata.datatype = class(read(userdata.reader,1));结束读者= userdata.reader;totalFrames = userdata.totalFrames;标签= userdata.label;batchesRead = userdata.batchesRead;如果isDataForTraining视频= readForTraining(读者,numFrames totalFrames);其他的视频= readForEvaluation(读者,userdata.datatype、numChannels numFrames, totalFrames);结束数据={视频标签};batchesRead = batchesRead + 1;用户数据。batchesRead = batchesRead;如果numFrames > totalFrames numBatches = 1;其他的numBatches =地板(totalFrames / numFrames);结束%完成标志设置为true,如果读者读过所有的帧或%如果是培训。做= batchesRead = = numBatches | | isDataForTraining;结束

函数视频= readForTraining(读者,numFrames totalFrames)如果numFrames > = totalFrames startIdx = 1;endIdx = totalFrames;其他的startIdx = randperm (totalFrames - numFrames + 1);startIdx = startIdx (1);endIdx = startIdx + numFrames - 1;结束视频=阅读(读者,[startIdx endIdx]);如果numFrames > totalFrames%添加更多的帧来填补在网络输入的大小。附加=装天花板(numFrames / totalFrames);视频= repmat(视频,1,1,1,额外的);视频=视频(:,:,:1:numFrames);结束结束

函数视频= readForEvaluation(读者,数据类型,numChannels numFrames, totalFrames) H = reader.Height;W = reader.Width;探路者= min ([numFrames totalFrames]);视频= 0 (H, W, numChannels,探路者,数据类型);frameIndex = 0;而hasFrame(读者)& & frameIndex < numFrames帧= readFrame(读者);frameIndex = frameIndex + 1;视频(:,:,:,frameIndex) =框架;结束如果frameIndex < numFrames视频=视频(:,:,:,1:frameIndex);附加=装天花板(numFrames / frameIndex);视频= repmat(视频,1,1,1,额外的);视频=视频(:,:,:1:numFrames);结束结束

函数标签= getLabel(文件名、类)文件夹= fileparts (string(文件名));[~,标签]= fileparts(文件夹);标签=分类(字符串(标签),字符串(类));结束

函数data = augmentVideo(数据)numClips =大小(数据,1);为2 = 1:numClips视频= {ii, 1}数据;% HxWxC深圳=大小(视频中,[1,2,3]);%每夹一个增强fcnaugmentFcn = augmentTransform(深圳);{二世,1}= augmentFcn数据(视频);结束结束

函数augmentFcn = augmentTransform(深圳)%随机图像翻转和规模。tform = randomAffine2d (“XReflection”,真的,“规模”1.1 [1]);tform溃败= affineOutputView(深圳,“BoundsStyle”,“CenterOutput”);augmentFcn = @(数据)augmentData(数据、tform溃败);函数data = augmentData(数据、tform溃败)数据= imwarp(数据、tform“OutputView”,溃败);结束结束

函数data = preprocessVideoClips(数据、信息)inputSize = info.InputSize (1:2);sizingOption = info.SizingOption;开关sizingOption情况下“调整”sizingFcn = @ (x) imresize (x, inputSize);情况下“randomcrop”sizingFcn = @ (x) cropVideo (x, @randomCropWindow2d inputSize);情况下“centercrop”sizingFcn = @ (x) cropVideo (x, @centerCropWindow2d inputSize);结束numClips =大小(数据,1);minValue = info.Statistics.Min;maxValue = info.Statistics.Max;meanValue = info.Statistics.Mean;stdValue = info.Statistics.StandardDeviation;minValue =重塑(minValue, 1, 1, 3);maxValue =重塑(maxValue, 1, 1, 3);meanValue =重塑(meanValue, 1, 1, 3);stdValue =重塑(stdValue, 1, 1, 3);为2 = 1:numClips视频= {ii, 1}数据;大小= sizingFcn(视频);%将单一的输入。大小=单(大小);%重新调节输入0和1之间。大小=重新调节(大小,0,1,InputMin = minValue InputMax = maxValue);%规范化使用平均值和标准偏差。大小=大小- meanValue;大小= resized. / stdValue;数据{ii, 1} =调整;结束函数outData = cropVideo(数据、cropFcn inputSize) imsz =大小(数据,[1,2]);cropWindow = cropFcn (imsz inputSize);numBatches =大小(数据,4);深圳= [inputSize、大小(数据,3),numBatches);outData = 0(深圳,“喜欢”、数据);为b = 1: numBatches outData (:,:,:, b) = imcrop(数据(:,:,:,b), cropWindow);结束结束结束

函数兆贝可= createMiniBatchQueue(数据存储、numOutputs params)如果参数个数。DispatchInBackground & & isempty (gcp (“nocreate”))%开始平行池,如果DispatchInBackground是真的,调度%在后台数据使用并行池。c = parcluster (“本地”);c。NumWorkers = params.NumWorkers;parpool (“本地”,params.NumWorkers);结束gcp (p =“nocreate”);如果~ isempty (p)数据存储= DispatchInBackgroundDatastore(数据存储,p.NumWorkers);结束inputFormat (1: numOutputs-1) =“SSCTB”;outputFormat =“CB”;numOutputs兆贝可= minibatchqueue(数据存储,…“MiniBatchSize”params.MiniBatchSize,…“MiniBatchFcn”@batchVideo,…“MiniBatchFormat”inputFormat, outputFormat]);结束

函数(视频、标签)= batchVideo(视频、标签)%批尺寸:5视频=猫(5、视频{:});%批尺寸:2标签=猫({}):2、标签;%功能维度:1= onehotencode标签(标签,1);结束

函数[gradientsRGB损失,acc, stateRGB] = modelGradients (slowFast dlRGB,海底)[dlYPredRGB stateRGB] =前进(slowFast dlRGB);dlYPred = squeezeIfNeeded (dlYPredRGB、海底);损失= crossentropy (dlYPred、海底);gradientsRGB = dlgradient(损失、slowFast.Learnables);%计算预测的准确性。[~,欧美]= max(海底,[],1);[~,YPred] = max (dlYPred [], 1);acc =收集(extractdata (sum(欧美= = YPred)。/元素个数(欧美)));结束

函数dlYPred = squeezeIfNeeded (dlYPred, Y)如果~ isequal(大小(Y),大小(dlYPred)) dlYPred =挤压(dlYPred);dlYPred = dlarray (dlYPred退去(Y));结束结束

函数[slowFast、速度,learnRate] = updateLearnables (slowFast、渐变参数、速度迭代)%确定学习速率使用cosine-annealing学习速率的时间表。learnRate = cosineAnnealingLearnRate(迭代、params);%对权重应用L2正规化。可学的= slowFast.Learnables;idx =可学的。参数= =“重量”;梯度(idx:) = dlupdate (@ (g, w) g + params.L2Regularization * w梯度(idx:),可学的(idx:));%更新使用个优化网络参数。[slowFast,速度]= sgdmupdate (slowFast、渐变速度,learnRate params.Momentum);结束

函数lr = cosineAnnealingLearnRate(迭代参数)如果迭代= =参数。NumIterations lr = params.MinLearningRate;返回;结束cosineNumIter = [0, params.CosineNumIterations];csum = cumsum (cosineNumIter);块=找到(迭代csum > = 1“第一”);cosineIter =迭代- csum (block - 1);annealingIteration =国防部(cosineIter cosineNumIter(块));cosineIteration = cosineNumIter(块);minR = params.MinLearningRate;maxR = params.MaxLearningRate;cosMult = 1 + cos(π* annealingIteration / cosineIteration);lr = minR + ((maxR - minR) * cosMult / 2);结束

函数欧美,cmat = aggregateConfusionMetric (cmat YPred)欧美=收集(extractdata(欧美));YPred =收集(extractdata (YPred));[m, n] =大小(cmat);cmat = cmat +满(稀疏(欧美YPred 1, m, n));结束

函数bestLoss = saveData (slowFast bestLoss,迭代,lossTrain params)如果迭代> =参数。SaveBestAfterIteration trainingLoss = extractdata(收集(lossTrain));如果trainingLoss < bestLoss bestLoss = trainingLoss;slowFast = gatherFromGPUToSave (slowFast);数据。Be年代tLoss = bestLoss; data.slowFast = slowFast; data.Params = params; save(params.ModelFilename,“数据”);结束结束结束

函数slowfast = gatherFromGPUToSave (slowfast)如果~ canUseGPU返回;结束slowfast。可学的= gatherValues (slowfast.Learnables);slowfast。状态= gatherValues (slowfast.State);函数台= gatherValues(台)为2 = 1:高度(台)台。{二}=收集价值(tbl.Value {2});结束结束结束

函数extractVideoScenes (groundTruthFolder、trainingFolder类)%如果视频场景已经提取,不需要下载%的数据集和提取视频场景。如果isfolder (trainingFolder) classFolders = fullfile (trainingFolder字符串(类));allClassFoldersFound = true;为2 = 1:元素个数(classFolders)如果~ isfolder (classFolders (ii) allClassFoldersFound = false;打破;结束结束如果allClassFoldersFound返回;结束结束如果~ isfolder (groundTruthFolder) mkdir (groundTruthFolder);结束downloadURL =“https://ssd.mathworks.com/金宝appsupportfiles/vision/data/videoClipsAndSceneLabels.zip”;文件名= fullfile (groundTruthFolder,“videoClipsAndSceneLabels.zip”);如果~存在(文件名,“文件”)disp (“下载视频剪辑和相应的标签”+ groundTruthFolder);websave(文件名,downloadURL);结束%解压缩下载文件夹的内容。解压缩(文件名,groundTruthFolder);labelDataFiles = dir (fullfile (groundTruthFolder“* _labelData.mat”));labelDataFiles = fullfile (groundTruthFolder, {labelDataFiles.name}”);numGtruth =元素个数(labelDataFiles);%加载标签数据信息和创建地面实况对象。gTruth = groundTruth.empty (numGtruth 0);为2 = 1:numGtruth ld =加载(labelDataFiles {2});videoFilename = fullfile (groundTruthFolder ld.videoFilename);gds = groundTruthDataSource (videoFilename);gTruth (ii) = groundTruth (gds, ld.labelDefs ld.labelData);结束%收集所有现场时间范围和相应的标签%使用sceneTimeRanges函数。[timeRanges, sceneLabels] = sceneTimeRanges (gTruth);%每个时间的子文件夹名称指定为现场标签的名字。foldernames = sceneLabels;%删除文件夹如果它已经存在。如果isfolder (trainingFolder)删除文件夹(trainingFolder,“年代”);结束%的视频文件被写入到指定的文件夹folderNames输入。writeVideoScenes (gTruth timeRanges、trainingFolder foldernames);结束

函数策划者= initializeTrainingProgressPlot (params)如果params.ProgressPlot%画出损失,训练精度,验证精度。图%损失情节次要情节(2,1,1)策划者。LossPlotter = animatedline;包含(“迭代”)ylabel (“损失”)%精度图次要情节(2,1,2)策划者。TrainAccPlotter = animatedline (“颜色”,“b”);传奇(“训练的准确性”,“位置”,“西北”);包含(“迭代”)ylabel (“准确性”)其他的策划者= [];结束结束

函数updateProgressPlot (params,策划者,时代,迭代,开始,lossTrain, accuracyTrain)如果params.ProgressPlot%更新培训的进展。D =持续时间(0,0,toc(开始),“格式”,“hh: mm: ss”);标题(plotters.LossPlotter.Parent,”时代:“+时代+”,过去:“+字符串(D));addpoints (plotters.LossPlotter、迭代、双(收集(extractdata (lossTrain))));addpoints (plotters.TrainAccPlotter迭代,accuracyTrain);drawnow结束结束

函数initializeVerboseOutput (params)如果参数个数。详细disp (”“)如果canUseGPU disp (“训练在GPU上。”)其他的disp (“训练在CPU上。”)结束gcp (p =“nocreate”);如果~ isempty (p) disp (并行集群上的“培训”+ p.Cluster。概要文件+“”。”)结束disp (”NumIterations:“+字符串(params.NumIterations));disp (”MiniBatchSize:“+字符串(params.MiniBatchSize));disp (“类:”+加入(字符串(params.Classes),”、“));disp (“| = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = |”)disp (| | |时代迭代时间| Mini-Batch | Mini-Batch学习| |基地训练时间|”)disp (“| | | (hh: mm: ss) | | | |率损失精度(hh: mm: ss) |”)disp (“| = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = |”)结束结束

函数displayVerboseOutputEveryEpoch(参数、启动、learnRate时代,迭代,…火车离站时刻表accTrain, lossTrain)如果参数个数。冗长的D =持续时间(0,0,toc(开始),“格式”,“hh: mm: ss”);火车离站时刻表=持续时间(0,0,火车离站时刻表,“格式”,“hh: mm: ss”);lossTrain =收集(extractdata (lossTrain));lossTrain =组成(“% .4f”,lossTrain);accTrain = composePadAccuracy (accTrain);learnRate =组成(“% .13f”,learnRate);disp (“|”+…垫(string(时代),5,“两个”)+“|”+…垫(字符串(迭代)9“两个”)+“|”+…垫(string (D) 12“两个”)+“|”+…垫(string (accTrain) 10“两个”)+“|”+…垫(string (lossTrain) 10“两个”)+“|”+…垫(string (learnRate), 13日“两个”)+“|”+…垫(string(火车离站时刻表)10“两个”)+“|”)结束函数acc = composePadAccuracy (acc) acc =组成(“% .2f”,100年acc *) +“%”;acc =垫(字符串(acc) 6“左”);结束结束

函数endVerboseOutput (params)如果参数个数。详细disp (“| = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = |”)结束结束