Neural Style Transfer Using Deep Learning

此示例使用：

打开实时脚本

This example shows how to apply the stylistic appearance of one image to the scene content of a second image using a pretrained VGG-19 network.

Load Data

加载样式图像和内容图像。该示例将独特的梵高绘画“星光之夜”用作样式图像，并将灯塔的照片作为内容图像。

styleImage = im2double(imread("starryNight.jpg"）;contentImage = imread（“lighthouse.png”);

Display the style image and content image as a montage.

imshow(imtile({styleImage,contentImage},BackgroundColor="w"）;

Load Feature Extraction Network

在此示例中，您使用经过修改的VGG-19深神经网络来提取各个层的内容和样式图像的特征。这些多层功能用于计算各自的内容和样式损失。网络使用合并损失生成风格化的转移图像。

要获得验证的VGG-19网络，请安装vgg19(Deep Learning Toolbox). If you do not have the required support packages installed, then the software provides a download link.

net = vgg19;

To make the VGG-19 network suitable for feature extraction, remove all of the fully connected layers from the network.

LastFeatUrelayerIdx = 38;layers = net.layers;层=层（1：lastFeatUrelayerIdx）;

VGG-19网络的最大池层会产生褪色效果。要降低褪色效果并增加梯度流量，请用平均池层替换所有最大池层[1].

forl = 1：lastFeatUrelayerIdx层=层（l）;ifisa（层，“ nnet.cnn.layer.maxpooling2dlayer”）layers（l）=平均泵2dlayer（layer.poolsize，stride = layer.stride，name = layer.name）;endend

Create a layer graph with the modified layers.

lgraph = layergraph（layers）;

可视化图中的特征提取网络。

情节（lgraph）标题（“特征提取网络”)

To train the network with a custom training loop and enable automatic differentiation, convert the layer graph to adlnetwork目的。

dlnet = dlnetwork(lgraph);

Preprocess Data

Resize the style image and content image to a smaller size for faster processing.

imageSize = [384,512]; styleImg = imresize(styleImage,imageSize); contentImg = imresize(contentImage,imageSize);

The pretrained VGG-19 network performs classification on a channel-wise mean subtracted image. Get the channel-wise mean from the image input layer, which is the first layer in the network.

imginputlayer = lgraph.layers（1）;MeanVggnet = imginputlayer.mean（1,1，:);

频道平均值的值适用于具有像素值在范围内的浮点数据类型的图像[0，255]。将样式图像和内容图像转换为数据类型singlewith range [0, 255]. Then, subtract the channel-wise mean from the style image and content image.

styleImg = rescale(single(styleImg),0,255) - meanVggNet; contentImg = rescale(single(contentImg),0,255) - meanVggNet;

初始化传输图像

传输图像的输出image as a result of style transfer. You can initialize the transfer image with a style image, content image, or any random image. Initialization with a style image or content image biases the style transfer process and produces a transfer image more similar to the input image. In contrast, initialization with white noise removes the bias but takes longer to converge on the stylized image. For better stylization and faster convergence, this example initializes the output transfer image as a weighted combination of the content image and a white noise image.

noiseRatio = 0.7; randImage = randi([-20,20],[imageSize 3]); transferImage = noiseRatio.*randImage + (1-noiseRatio).*contentImg;

Define Loss Functions and Style Transfer Parameters

内容损失

内容丢失的目标是特点res of the transfer image match the features of the content image. The content loss is calculated as the mean squared difference between content image features and transfer image features for each content feature layer[1]. $Y_{}^{ˆ}$ is the predicted feature map for the transfer image and $Y$ 是内容图像的预测特征映射。 $W_{c}^{l}$ 是内容层的重量 $l^{t h}$ layer. $H, W, C$ 分别是特征图的高度，宽度和通道。

$L_{c o n t e n t} = \sum_{l} W_{c}^{l} \times \frac{1}{H W C} \sum_{i, j} ({Y_{}^{ˆ}}_{i, j}^{l} - Y_{i, j}^{l})^{2}$

指定内容特征提取层名称。从这些层中提取的功能用于计算内容损失。在VGG-19网络中，使用较深层的功能而不是浅层层的功能，培训更有效。因此，将内容特征提取层指定为第四卷积层。

styletRansferoptions.contentFeatUrelayErnames ="conv4_2";

指定内容特征提取层的权重。

styleTransferOptions.contentFeatureLayerWeights = 1;

样式损失

样式损失的目的是使转移图像的纹理与样式图像的纹理相匹配。图像的样式表示为革兰氏矩阵。因此，样式损失被计算为样式图像的革兰氏矩阵与转移图像的革兰氏矩阵之间的平方平方差[1]. $Z$ 和 $Z_{}^{ˆ}$ are the predicted feature maps for the style and transfer image, respectively. $G_{Z}$ 和 $G_{Z_{}^{ˆ}}$ 分别是样式功能和传输功能的克矩阵。 $W_{s}^{l}$ is the style layer weight for the $l^{t h}$ style layer.

$G_{Z_{}^{ˆ}} = \sum_{i, j} {Z_{}^{ˆ}}_{i, j} \times {Z_{}^{ˆ}}_{j, i}$

$G_{Z} = \sum_{i, j} Z_{i, j} \times Z_{j, i}$

$L_{s t y l e} = \sum_{l} W_{s}^{l} \times \frac{1}{(2 H W C)^{2}} \sum (G_{Z_{}^{ˆ}}^{l} - G_{Z}^{l})^{2}$

Specify the names of the style feature extraction layers. The features extracted from these layers are used to calculate style loss.

StyletRansferoptions.StyleFeatUrelayErnames = [“ conv1_1”,“ conv2_1”,“ conv3_1”,"conv4_1","conv5_1"];

Specify the weights of the style feature extraction layers. Specify small weights for simple style images and increase the weights for complex style images.

StyletRansferoptions.StyleFeatUrelayerWeaights = [0.5,1.0,1.5,3.0,4.0];

总体损耗

The total loss is a weighted combination of content loss and style loss. $α$ 和 $β$ 是内容损失和样式损失的重量因素。

$L_{t o t a l} = α \times L_{c o n t e n t} + β \times L_{s t y l e}$

指定重量因素alpha和beta内容损失和样式损失。比率alphatobeta应大约为1E-3或1E-4[1].

styletRansferoptions.alpha = 1;styletRansferoptions.beta = 1e3;

Specify Training Options

Train for 2500 iterations.

计算= 2500;

Specify options for Adam optimization. Set the learning rate to 2 for faster convergence. You can experiment with the learning rate by observing your output image and losses. Initialize the trailing average gradient and trailing average gradient-square decay rates with[].

learningRate = 2; trailingAvg = []; trailingAvgSq = [];

Train the Network

Convert the style image, content image, and transfer image todlarray(Deep Learning Toolbox)具有基础类型的对象single和dimension labels "SSC".

dlStyle = dlarray(styleImg,“ SSC”); dlContent = dlarray(contentImg,“ SSC”); dlTransfer = dlarray(transferImage,“ SSC”);

Train on a GPU if one is available. Using a GPU requires Parallel Computing Toolbox™ and a CUDA® enabled NVIDIA® GPU. For more information, see释放的G金宝appPU支持(Parallel Computing Toolbox). For GPU training, convert the data into agpuArray.

ifcanUseGPU dlContent = gpuArray(dlContent); dlStyle = gpuArray(dlStyle); dlTransfer = gpuArray(dlTransfer);end

Extract the content features from the content image.

numContentFeatUrelayers = numel（styletRansferoptions.contentFeatUrelayerNames）;contentFeatures =单元格（1，numContentFeatUrelayers）;[contentFeatures {：}] = forward（dlnet，dlcontent，outputs = styletRansferoptions.contentFeatUrelayerNames）;

Extract the style features from the style image.

numStyleFeatUrelayers = numel（styletRansferoptions.StyleFeatUrelayErnames）;stylefeatures =单元格（1，numstylefeaturelayers）;[stylefeatures {：}] = forward（dlnet，dlStyle，outputs = styletRansferoptions.StyleFeatUreLayerNames）;

Train the model using a custom training loop. For each iteration:

Calculate the content loss and style loss using the features of the content image, style image, and transfer image. To calculate the loss and gradients, use the helper functionimageGradients(defined in theSupporting Functions此示例的部分）。
Update the transfer image using theAdamupdate(Deep Learning Toolbox)功能.
选择最佳样式传输图像作为最终输出图像。

图最小值= inf;foriteration = 1:numIterations％使用DLFEVAL评估转移图像梯度和状态% imageGradients function listed at the end of the example[grad,losses] = dlfeval(@imageGradients,dlnet,dlTransfer,contentFeatures,styleFeatures,styleTransferOptions); [dlTransfer,trailingAvg,trailingAvgSq] = adamupdate(dlTransfer,grad,trailingAvg,trailingAvgSq,iteration,learningRate);iflosses.totalLoss < minimumLoss minimumLoss = losses.totalLoss; dlOutput = dlTransfer;end% Display the transfer image on the first iteration and after every 50% iterations. The postprocessing steps are described in the "Postprocess％转移图像显示“此示例的部分”ifmod(iteration,50) == 0 || (iteration == 1) transferImage = gather(extractdata(dlTransfer)); transferImage = transferImage + meanVggNet; transferImage = uint8(transferImage); transferImage = imresize(transferImage,size(contentImage,[1 2])); image(transferImage) title(["Transfer Image After Iteration ",num2str(iteration)]) axis离开imagedrawnowendend

Postprocess Transfer Image for Display

获取更新的传输图像。

transferImage = gather(extractdata(dlOutput));

Add the network-trained mean to the transfer image.

transferImage = transferImage + meanVggNet;

某些像素值可以超过内容和样式图像的原始范围[0，255]。您可以通过将数据类型转换为[0，255]范围[0，255]uint8.

TransferImage = UINT8（转移图）;

Resize the transfer image to the original size of the content image.

TransferImage = Imresize（转移图像，大小（contentImage，[1 2]））;

在蒙太奇中显示内容图像，传输图像和样式图像。

imshow(imtile({contentImage,transferImage,styleImage},...GridSize=[1 3],BackgroundColor="w"）;

Supporting Functions

Calculate Image Loss and Gradients

TheimageGradientshelper function returns the loss and gradients using features of the content image, style image, and transfer image.

功能[渐变，损失] = imageGradients（dlnet，dltransfer，contentfeatures，stylefeatures，params）% Initialize transfer image feature containersnumContentFeatUrelayers = numel（params.contentFeatUrelayerNames）;numStyleFeatUrelayers = numel（params.StyleFeatUrelayerNames）;TransferContentFeatures = Cell（1，NumContentFeatUrelayers）;transferstylefeatures =单元格（1，numstylefeaturelayers）;％提取转移图像的内容特征[transferContentFeatures{:}] = forward(dlnet,dlTransfer,Outputs=params.contentFeatureLayerNames);％提取样式特征的转移图像[transferStyleFeatures{:}] = forward(dlnet,dlTransfer,Outputs=params.styleFeatureLayerNames);% Calculate content losscloss = ContentLoss（TransferContentFeatures，ContentFeatures，params.contentFeatUrelayerWeaights）;％计算样式损失sLoss = styleLoss(transferStyleFeatures,styleFeatures,params.styleFeatureLayerWeights);% Calculate final loss as weighted combination of content and style loss损失=（params.alpha * closs） +（params.beta * sloss）;% Calculate gradient with respect to transfer imagegradients = dlgradient(loss,dlTransfer);％提取各种损失losses.totalLoss = gather(extractdata(loss)); losses.contentLoss = gather(extractdata(cLoss)); losses.styleLoss = gather(extractdata(sLoss));end

计算内容损失

Thecontentlosshelper function calculates the weighted mean squared difference between the content image features and the transfer image features.

功能loss = contentLoss(transferContentFeatures,contentFeatures,contentWeights) loss = 0;fori = 1：numel（contentFeatures）temp = 0.5。“全部”); loss = loss + (contentWeights(i)*temp);endend

计算样式损失

TheStylelosshelper function calculates the weighted mean squared difference between the Gram matrix of the style image features and the Gram matrix of the transfer image features.

功能损失= styleloss（transferstylefeatures，stylefeatures，styleWeights）损失= 0;fori = 1：numel（stylefeatures）tsf = transferstylefeatures {1，i};sf = stylefeatures {1，i};[h，w，c] = size（sf）;gramstyle =计算grammatrix（sf）;gramTransfer =计算Grammatrix（TSF）;sloss =平均值（（gramtransfer -gramstyle）。^2，“全部”) / ((h*w*c)^2); loss = loss + (styleWeights(i)*sLoss);endend

Calculate Gram Matrix

ThecalculateGramMatrixhelper function is used by theStylelosshelper function to calculate the Gram matrix of a feature map.

功能gramMatrix = calculateGramMatrix(featureMap) [H,W,C] = size(featureMap); reshapedFeatures = reshape(featureMap,H*W,C); gramMatrix = reshapedFeatures' * reshapedFeatures;end

参考

[1]Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. "A Neural Algorithm of Artistic Style." Preprint, submitted September 2, 2015. https://arxiv.org/abs/1508.06576