This post is fromOge Marques, PhD以及FAU的工程与计算机科学教授。OGE是一个Sigma Xi Distinguished Speaker，，，，book author，，，，andAAAS LESHNER研究员。他也恰好是Matlab狂热者，并且在他的教室里使用Matlab已有20多年了。您可以在Twitter上关注他（@ProfessorOge）。在这篇博客文章中，OGE将介绍如何使用Python和Matlab进行计算机视觉示例进行深入学习。

深度学习(DL)技术已经改变了5ld of computer vision significantly during the last decade, providing state-of-the-art solutions for classical tasks (e.g.,object detectionand图像分类) and opening the doors for solving challenging new problems, such asimage-to-image translationandvisual question answering (VQA)。

DL在计算机视觉及相关领域的成功和普及（例如，医学图像分析）在很大程度上是通过在The The The The Pression，Apps和Frameworks的可用性中促进的PythonandMATLABecosystems.

在这篇博客文章中，我将展示您的团队如何有效地使用MATLAB和PYTHON，并提供易于遵循的食谱，在使用深度学习构建计算机视觉解决方案时，您应该允许您利用“两全其美”。金宝搏官方网站

背景

Pythonis a programming language created by Guido van Rossum in the early 1990s. It has been adopted by many data scientists and machine/deep learning researchers thanks to popular packages (e.g.,Scikit-Learn）和框架（例如，凯拉斯，，，，TensorFlow，，，，Pytorch）。

MATLAB是一种编程和科学计算平台used to analyze data, develop algorithms, and create models in a variety of fields of science and engineering. It has a successful history of widespread adoption by engineers and researchers in industry and academia. It features many specialized工具箱which encapsulate relevant algorithms, interactive tools, and rich examples in areas such as机器学习，，，，deep learning，，，，image processing，，，，andcomputer vision(to mention but a few). MATLAB also has a vibrant用户社区who contribute additional functionality (including应用and entire工具箱) and a growing presence in popular code-sharing repositories such asGitHub。

根据我的个人经验，我在不同的研究项目，课程，训练营和出版物中都使用了MATLAB（25年）和Python（不到十年），主要是在图像处理/分析的背景下，以及（最近）数据科学，机器学习和深度学习。

I have also worked with multidisciplinary teams who adopt a variety of tools and are well-versed in diverse skill sets. I know how important it is to promote and facilitate the adoption of a streamlined and well-documented deep learning workflow. I am also a strong proponent of always using the best available tools to get the job done in the best possible way. Fortunately, you can use the two languages together, which we will show next.

Context and scope

The interoperability of MATLAB and Python has been extensively documented in视频，，，，webinars，，，，blog posts, and theMATLAB官方文档。学习时这些资源可能非常有价值how to call Python scripts from MATLAB and vice-versa。的一些主要原因从Python打电话给MATLAB可以是由需要的动机来激励的：

1. Promote code integration among team members and collaborators using different frameworks and tools.
2. 利用功能仅在MATLAB中可用，例如应用and toolboxes (including third-party ones contributed by the MATLAB community).

In this blog post, I focus onintegrating MATLAB apps into a Python deep learning workflow for computer vision and image analysis tasks，，，，with emphasis on the data preparation stage of the traditional deep learning workflow (Figure 1). More specifically, I show how your team can leverage the rich interactive capabilities of selected MATLAB apps to prepare, label, annotate, and preprocess your databeforeusing it as the input to your neural network – and everything else that follows in the traditional deep learning pipeline.

Figure 1: Basic deep learning workflow.

I will assume that: (1) you have a deep learning pipeline for computer vision in Python that you plan to adapt and reuse for a new (set of) task(s); and (2) the images associated with the new task(s) will require interactive actions, such as annotation, labeling, and segmentation.

The basic recipe

假设你有MATLAB installed and configured in your machine以及您最喜欢的Python设置（例如，使用Jupyter笔记本电脑），从Python脚本调用MATLAB是一个简单的过程，其主要步骤是：

1. (In MATLAB)Install theMATLAB Engine API for Python，，，，which provides a Python package called matlab that allows you to call MATLAB functions and exchange data between Python and MATLAB.
2. (In Python)Configure paths and working directory.
3. (In Python)在背景中启动新的MATLAB过程：

进口Matlab.Engine eng = matlab.engine.start_matlab('-desktop')

4. (In Python)Set up your variables (e.g., path to image folders).
5. (In Python)Call a MATLAB app of your choice (e.g., Image Labeler app).
6. (In MATLAB)（交互式）与所选的应用程序进行工作，并将结果导出到工作空间中的变量。
7. (In Python)保存其余工作流程所需的变量，例如图像文件名和关联的标签（及其边界框）。
8. (In Python)使用variables as needed, e.g., processing tabular data using pandas and using image-related labels as ground truth.
9. 在工作流程中根据需要重复步骤3至7次。
10. (In Python)Quit the MATLAB engine:

eng.exit()

An example

Here is an例子关于如何在医学图像分析范围内使用Python和Matlab进行两个不同任务（使用深度学习）：皮肤病变细分和（医学）图像（ROI）标签。

尽管它们之间存在差异，但每个任务都遵循前面提出的相同基本食谱。接下来描述每种情况的细节。

任务A：皮肤病变细分

任务：Given a dataset of images containing skin lesions, we want to build a deep learning solution for segmenting each image, i.e., classifying each pixel as belonging to either the lesion (foreground) or the rest of the image (background).

问题：In order to train and validate a deep neural network for image segmentation, we need to input both the images as well as the segmentation masks (Figure 2), which are essentially binary images where foreground pixels (in this case corresponding to the lesion) are labeled white and background pixels are marked as black. The job of the network is to learn the segmentation masks for new images.

Figure 2: Skin lesion segmentation: input image (left); binary segmentation mask (center); segmented image, with green contour outlining the lesion area (right).

基本工作流程通常包括使用卷积网络体系结构，例如U-netand its variations, for which there are multiple examples of implementation inPythonandMATLAB。A crucial component of the solution, however, is the manual creation of the binary masks needed for training and validation. Except for a few publicly available datasets, this time-consuming and specialized task must be performed using a powerful interactive tool.

解决方案：使用MATLAB图像细分器应用程序to create the binary masks and leverage the existing (Python, for the sake of this example) workflow for everything else. Image Segmenter allows you to create masks manually and provides several (semi-)automatic techniques to speed up the process and refine the results (Figures 3 and 4). Both the final segmentation mask image and the segmented version of the original image can be exported to the MATLAB workspace and/or saved to disk.

图3：图像分段应用程序：加载包含皮肤病变的图像。

图4：图像细分器应用程序：在选择了一些前景控制点（绿色）和单个背景控制点（红色）后应用图形切割算法的结果。面具出现在原始图像的顶部。

任务B：（医疗）图像（ROI）标签

任务：In a similar context to Task A, we want to build a deep learning solution for detecting regions of interest (ROIs) in each image, i.e., placing a boundary around each relevant region in the image. The most common ROI will be a lesion; other possible ROIs could include stickers, ruler markers, water bubbles, ink marks, and other artifacts.

问题：To train and validate a deep neural network for ROI/object detection, we need to input both the images as well as the labels and coordinates of the relevant ROIs, which can be expressed as rectangles (most common), polygons, or pixel-based masks (similarly to the masks used in segmentation). The job of the network is to learn the location and labels of the relevant ROIs for new images.

Once again, similarly to what we saw in Task A, a crucial component of the solution is the manual creation of the ROIs (polygons and labels) needed for training and validation. Except for a few publicly available datasets, this time-consuming and specialized task must be performed using a powerful interactive tool.

解决方案：使用MATLAB图像标签应用程序to create and label the ROIs and leverage the existing workflow for everything else. Image Labeler allows you to create ROI labels of different shape, assign them different names and colors, and provides several algorithms to help automate and speed up the process and refine the results (Figure 5). The resulting ROIs can be exported to the MATLAB workspace and subsequently used as variables in your Python code (see例子on GitHubfor details).

Figure 5: Image Labeler app in the context of dermoscopic images containing artifacts. The selected image contains two rectangular ROIs, labeled aslesionand贴纸。

关键要点

Deep Learning projects are often collaborative endeavors that require using the best tools for the job, enabling effective code integration, development, and testing strategies, promoting communication, and ensuring reproducibility of code. Your team can (and should) leverage the best of MATLAB and Python while developing your deep learning projects. In this blog post I have shown how to use Python and MATLAB together for a few tasks related to computer vision and medical image analysis problems.

Python和Matlab的集成大大超出了此博客文章的范围；查看下面列出的资源以获取更多信息。

Learn more about it

This blog post was inspired byrecent blog posts by Lucas García以及一系列精彩视频希瑟·戈尔（Heather Gorr），Yann DeBray和同事。我强烈建议您关注他们并查看他们非常有用的示例和教程。如果您对深度学习工作流程的其他方面感兴趣，那么这些是我的一些博客文章：

• Discussthe entire process (including often forgotten steps) in greater detail;
• Showhow to use a low-code app in MATLAB, the Deep Network Designer, for...;and
• Teach youhow to manage and track multiple deep learning experiments具有不同的网络体系结构，超参数和其他选项。去看一下！