This example shows how to train a network that classifies handwritten digits with a custom learning rate schedule.

You can train most types of neural networks using thetrainNetworkandtrainingOptionsfunctions. If thetrainingOptionsfunction does not provide the options you need (for example, a custom learning rate schedule), then you can define your own custom training loop usingdlarrayanddlnetworkobjects for automatic differentiation. For an example showing how to retrain a pretrained deep learning network using thetrainNetworkfunction, seeTransfer Learning Using Pretrained Network.

Training a deep neural network is an optimization task. By considering a neural network as a function $$f(X;\theta )$$, where $$X$$is the network input, and $$\theta$$is the set of learnable parameters, you can optimize $$\theta$$so that it minimizes some loss value based on the training data. For example, optimize the learnable parameters $$\theta$$such that for a given inputs $$X$$with a corresponding targets $$T$$, they minimize the error between the predictions $$Y=f(X;\theta )$$and $$T$$.

取决于使用的损失函数类型的任务. For example:

You can optimize the objective using gradient descent: minimize the loss $$L$$by iteratively updating the learnable parameters $$\theta$$通过步骤使用毕业生的最低ients of the loss with respect to the learnable parameters. Gradient descent algorithms typically update the learnable parameters by using a variant of an update step of the form $${\theta }_{t+1}={\theta }_t-\rho \nabla L$$, where $$t$$is the iteration number, $$\rho$$is the learning rate, and $$\nabla L$$denotes the gradients (the derivatives of the loss with respect to the learnable parameters).

This example trains a network to classify handwritten digits with thetime-based decaylearning rate schedule: for each iteration, the solver uses the learning rate given by ${\rho }_{\mathit{t}}=\frac{{\rho }_0}{1+\mathit{k}\mathit{t}}$, wheretis the iteration number, $${\rho }_0$$is the initial learning rate, andkis the decay.

dataFolder = fullfile(toolboxdir("nnet"),"nndemos","nndatasets","DigitDataset"); imds = imageDatastore(dataFolder,...IncludeSubfolders=true,....LabelSource="foldernames");

[imdsTrain,imdsValidation] = splitEachLabel(imds,0.9,"randomize");

inputSize = [28 28 1]; pixelRange = [-5 5]; imageAugmenter = imageDataAugmenter(...RandXTranslation=pixelRange,...RandYTranslation=pixelRange); augimdsTrain = augmentedImageDatastore(inputSize(1:2),imdsTrain,DataAugmentation=imageAugmenter);

augimdsValidation = augmentedImageDatastore(inputSize(1:2),imdsValidation);

classes = categories(imdsTrain.Labels); numClasses = numel(classes);

layers = [ imageInputLayer(inputSize,Normalization="none") convolution2dLayer(5,20,Padding="same") batchNormalizationLayer reluLayer convolution2dLayer(3,20,Padding="same") batchNormalizationLayer reluLayer convolution2dLayer(3,20,Padding="same") batchNormalizationLayer reluLayer fullyConnectedLayer(numClasses) softmaxLayer];

net = dlnetwork(layers)

net = dlnetwork with properties: Layers: [12×1 nnet.cnn.layer.Layer] Connections: [11×2 table] Learnables: [14×3 table] State: [6×3 table] InputNames: {'imageinput'} OutputNames: {'softmax'} Initialized: 1 View summary with summary.

numEpochs = 10; miniBatchSize = 128;

initialLearnRate = 0.01; decay = 0.01; momentum = 0.9;

mbq = minibatchqueue(augimdsTrain,...MiniBatchSize=miniBatchSize,...MiniBatchFcn=@preprocessMiniBatch,...MiniBatchFormat=["SSCB"""]);

velocity = [];

numObservationsTrain = numel(imdsTrain.Files); numIterationsPerEpoch = ceil(numObservationsTrain / miniBatchSize); numIterations = numEpochs * numIterationsPerEpoch;

monitor = trainingProgressMonitor(Metrics="Loss",Info=["Epoch","LearnRate"],XLabel="Iteration");

epoch = 0; iteration = 0;% Loop over epochs.whileepoch < numEpochs && ~monitor.Stop epoch = epoch + 1;% Shuffle data.shuffle(mbq);% Loop over mini-batches.whilehasdata(mbq) && ~monitor.Stop iteration = iteration + 1;% Read mini-batch of data.[X,T] = next(mbq);% Evaluate the model gradients, state, and loss using dlfeval and the% modelLoss function and update the network state.[loss,gradients,state] = dlfeval(@modelLoss,net,X,T); net.State = state;% Determine learning rate for time-based decay learning rate schedule.learnRate = initialLearnRate/(1 + decay*iteration);% Update the network parameters using the SGDM optimizer.[net,velocity] = sgdmupdate(net,gradients,velocity,learnRate,momentum);% Update the training progress monitor.recordMetrics(monitor,iteration,Loss=loss); updateInfo(monitor,Epoch=epoch,LearnRate=learnRate); monitor.Progress = 100 * iteration/numIterations;endend

numOutputs = 1; mbqTest = minibatchqueue(augimdsValidation,numOutputs,...MiniBatchSize=miniBatchSize,...MiniBatchFcn=@preprocessMiniBatchPredictors,...MiniBatchFormat="SSCB");

YTest = modelPredictions(net,mbqTest,classes);

TTest = imdsValidation.Labels; accuracy = mean(TTest == YTest)

accuracy = 0.9750

figure confusionchart(TTest,YTest)

function[loss,gradients,state] = modelLoss(net,X,T)% Forward data through network.[Y,state] = forward(net,X);% Calculate cross-entropy loss.loss = crossentropy(Y,T);% Calculate gradients of loss with respect to learnable parameters.gradients = dlgradient(loss,net.Learnables);end

functionY = modelPredictions(net,mbq,classes) Y = [];% Loop over mini-batches.whilehasdata(mbq) X = next(mbq);% Make prediction.scores = predict(net,X);% Decode labels and append to output.labels = onehotdecode(scores,classes,1)'; Y = [Y; labels];endend

function[X,T] = preprocessMiniBatch(dataX,dataT)% Preprocess predictors.X = preprocessMiniBatchPredictors(dataX);% Extract label data from cell and concatenate.T = cat(2,dataT{1:end});% One-hot encode labels.T = onehotencode(T,1);end

functionX = preprocessMiniBatchPredictors(dataX)% Concatenate.X = cat(4,dataX{1:end});end