Video Labeler

Label video for computer vision applications

expand all in page

Description

的Video Labelerapp enables you to label ground truth data in a video, in an image sequence, or from a custom data source reader. Using the app, you can:

Define rectangular regions of interest (ROI) labels, polyline ROI labels, pixel ROI labels, and scene labels. Use these labels to interactively label your ground truth data.
Use built-in detection or tracking algorithms to label your ground truth data.
Write, import, and use your own custom automation algorithm to automatically label ground truth. SeeCreate Automation Algorithm for Labeling.
Evaluate the performance of your label automation algorithms using a visual summary. SeeView Summary of Ground Truth Labels.
Export the labeled ground truth as agroundTruth对象。您可以使用此对象系统verification or for training an object detector or semantic segmentation network. SeeTraining Data for Object Detection and Semantic Segmentation.

To learn more about this app, seeGet Started with the Video Labeler.

Open the Video Labeler App

MATLAB^®Toolstrip: On theAppstab, underImage Processing and Computer Vision, click the app icon.
MATLAB command prompt: EntervideoLabeler.

Examples

Get Started with the Video Labeler

Programmatic Use

expand all

`videoLabeler`

videoLabeleropens a new session of the app, enabling you to label ground truth data in a video or image sequence.

`videoLabeler(videoFileName)`

videoLabeler(videoFileName)打开应用程序,加载输入视频。的video file must have an extension supported byVideoReader.

Example:videoLabeler('vipmen.avi')

`videoLabeler(imageSeqFolder)`

videoLabeler(imageSeqFolder)opens the app and loads the image sequence from the input folder. Animage sequenceis an ordered set of images that resemble a video.

imageSeqFoldermust be a string scalar or character vector that specifies the folder containing the image files. The image files must have extensions supported byimformatsand are loaded in the order returned by thedirfunction.

`videoLabeler(imageSeqFolder,timestamps)`

videoLabeler(imageSeqFolder,timestamps)opens the app and loads a sequence of images with their corresponding timestamps.timestampsmust be adurationvector of the same length as the number of images in the sequence.

For example, load a sequence of images and their corresponding timestamps into the app.

imageDir = fullfile(toolboxdir('vision'),'visiondata','NewTsukuba'); timeStamps = seconds(1:150); videoLabeler(imageDir,timeStamps)

`videoLabeler(gtSource)`

videoLabeler(gtSource)opens the app and loads the data source and corresponding timestamps from agroundTruthDataSourceobject,gtSource. To generate this object for a custom data source, you can specify a custom reader function. For details, seeUse Custom Image Source Reader for Labeling.

`videoLabeler(sessionFile)`

videoLabeler(sessionFile)opens the app and loads a saved app session,sessionFile. ThesessionFileinput contains the path and file name. The MAT-file thatsessionFilepoints to contains the saved session.

`videoLabeler(gTruth)`

videoLabeler(gTruth)opens the app and loads agroundTruth对象。You do not need to add a video before loading the object. The ground truth object data source must be an image sequence, video, or a custom data source.

Limitations

的built-in automation algorithms support the automation of rectangular ROI labels only. When you select a built-in algorithm and clickAutomate、现场标签、像素ROI标签,折线ROI实验室els, sublabels, and attributes are not imported into the automation session. To automate the labeling of these features, create a custom automation algorithm. SeeCreate Automation Algorithm for Labeling.
Pixel ROI labels do not support sublabels or attributes.
的Label Summary window does not support sublabels or attributes

More About

expand all

ROI Labels, Sublabels, and Attributes

On the left side of the app, theROI Labelspane contains the region of interest (ROI) label definitions that you can mark on the frames. You can create label definitions directly from this pane. Alternatively, you can create label definitions programmatically by using alabelDefinitionCreatorobject and then import these label definitions into an app session.

的app supports the definition of ROI labels, sublabels, and attributes.

Selecting ROI Label and Drawing Tool for Your Application

You can use labeled data to validate or train algorithms such as image classifiers, object detectors, and semantic and instance segmentation networks. Consider your application when choosing a labeling drawing tool to create ROI labels. The figure below shows labeling techniques for four different applications.

Application	Label Types	Example
Image Classification	N/A	Boat, Plane
Object Detection	Line Rectangle Projected Rectangle
Semantic Segmentation	Pixel label Polygon
Instance Segmentation	Polygon

ROI Labels

AnROI labelis a label that corresponds to a region of interest (ROI) in a signal frame. The table describes the supported label types.

ROI Label	Description	Scene
`Rectangle`	Draw rectangular ROI labels (bounding boxes) around objects.	Vehicles, pedestrians, road signs
`Projected cuboid`	Draw cuboidal ROI labels (3-D bounding boxes).
`Line`	Draw linear ROI labels to represent lines. To draw a polyline ROI, use two or more points.	Lane boundaries, guard rails, road curbs
`Pixel label`	Assign labels to pixels for semantic segmentation. You can label pixels manually using polygons, brushes, or flood fill. For more on pixel labeling, seeLabel Pixels for Semantic Segmentation.	Vehicles, road surface, trees, pavement
`Polygon`	Draw polygon ROI labels. You can label distinct instances of the same class. For more information on drawing polygon ROI labels for instance and semantic segmentation networks, seeLabel Objects Using Polygons	Vehicles, road surface, trees, pavement

ROI Sublabels

AnROI sublabelis an ROI label that belongs to a parent label. Use ROI sublabels to provide a greater level of detail about the ROIs in your labeled ground truth data. For example, avehiclelabel might containheadlight,licensePlate, andwheelsublabels. You can create sublabels only for rectangular and polyline labels. For more details about sublabels, seeUse Sublabels and Attributes to Label Ground Truth Data.

ROI Attributes

AnROI attributespecifies additional information about an ROI label or sublabel. For example, in a driving scene, attributes might include the type or color of a vehicle. The table describes the supported attribute types.

Attribute Type	Sample Attribute Definition	Sample Default Values
`Numeric Value`
`String`
`Logical`
`List`

For more details about attributes, seeUse Sublabels and Attributes to Label Ground Truth Data.

Tips

To avoid having to relabel ground truth with new labels, organize the labeling scheme you want to use before marking your ground truth.

Algorithms

expand all

You can use label automation algorithms to speed up labeling within the app. To create your own label automation algorithm to use within the app, seeCreate Automation Algorithm for Labeling. You can also use one of the built-in algorithms by following these steps:

Import the data you want to label, and create at least one label definition.
On the app toolstrip, clickSelect Algorithmand select one of the built-in automation algorithms.
ClickAutomate, and then follow the automation instructions in the right pane of the automation window.

ACF People Detector

Detect and label people using aggregate channel features (ACF). This algorithm is based on thepeopleDetectorACFfunction. To use this algorithm, you must define at least one rectangle ROI label. You do not need to draw any ROI labels.

To help improve the algorithm results, first clickSettings. You can change any of these settings.

的pretrained people detector model that the algorithm uses — The'inria-100x41'model was trained using the INRIA person data set. The'caltech-50x21'model was trained using the Caltech Pedestrian data set.
的overlap ratio threshold, from 0 to 1, for detecting people — When rectangle ROIs overlap by more than this threshold, the algorithm discards one of the ROIs.
的classification score threshold for detecting people — Increase the score to increase the prediction confidence of the algorithm. Rectangles with scores below this threshold are discarded.

Point Tracker

Track and label one or more rectangle ROI labels over short intervals by using the Kanade-Lucas-Tomasi (KLT) algorithm. This algorithm is based on thevision.PointTrackerSystem object™. To use this algorithm, you must define at least one rectangle ROI label, but you do not need to draw any ROI labels.

To change the feature detector used to obtain the initial points for tracking, clickSettings. This table shows the feature detector options.

Feature Detector	Description	Equivalent Function
`Minimum Eigen Value`	Detect corners by using the minimum eigenvalue algorithm.	`detectMinEigenFeatures`
`Harris`	Detect corners by using the Harris–Stephens algorithm.	`detectHarrisFeatures`
`FAST`	Detect corners by using the features from accelerated segment test (FAST) algorithm.	`detectFASTFeatures`
`BRISK`	Detect features by using the binary robust invariant scalable keypoints (BRISK) algorithm.	`detectBRISKFeatures`
`KAZE`	Detect features by using nonlinear diffusion to construct a scale space of an image, and then detecting multiscale corner features (KAZE features) from that scale space.	`detectKAZEFeatures`
`SURF`	Detect blob features by using the speeded-up robust features (SURF) algorithm.	`detectSURFFeatures`
`MSER`	Detect regions by using the maximally stable extremal regions (MSER) algorithm.	`detectMSERFeatures`

Temporal Interpolator

Estimate rectangle ROIs between frames by interpolating the ROI locations across the time range. To use this algorithm, you must draw a rectangle ROI on a minimum of two frames: one at the beginning of the interval and one at the end of the interval. The interpolation algorithm estimates and draws ROIs in the intermediate frames.

Consider a video with 10 frames. The first frame has a rectangle ROI centered at [5, 5]. The 10th frame has a rectangle ROI centered at [25, 25]. At each frame, the algorithm moves the ROI 2 pixels in thex-direction and 2 pixels in they-direction. Therefore, the algorithm centers the ROI at [7, 7] in the second frame, [9, 9] in the third frame, and so on, up to [23, 23] in the second-to-last frame.

ACF Vehicle Detector(requiresAutomated Driving Toolbox)

Detect and label vehicles using aggregate channel features (ACF). This algorithm is based on thevehicleDetectorACF(Automated Driving Toolbox)function. To use this algorithm, you must define at least one rectangle ROI label. You do not need to draw any ROI labels.

To help improve the algorithm results, first clickSettings. You can change any of these settings.

的pretrained vehicle detector model that the algorithm uses — The'full-view'model was trained using unoccluded images of the front, rear, left, and right sides of vehicles. The'front-rear-view'model was trained using images of only the front and rear sides of the vehicle.
的overlap ratio threshold, from 0 to 1, for detecting vehicles — When rectangle ROIs overlap by more than this threshold, the algorithm discards one of the ROIs.
分类分阈值检测vehicles — Increase the score to increase the prediction confidence of the algorithm. Rectangles with scores below this threshold are discarded.

You can also configure the detector with a calibrated monocular camera by importing amonoCamera(Automated Driving Toolbox)object into the MATLAB workspace. Specify the length and width ranges of the vehicle in world units, such as meters.

Lane Boundary Detector(requiresAutomated Driving Toolbox)

Detect and label lane boundaries using an estimated bird’s-eye-view projected image. To use this algorithm, you must define at least one line ROI label. You do not need to draw any ROI labels. To detect lane boundaries, the algorithm follows these steps:

It makes an initial guess at the placement of the lane boundaries in the image.
It transforms the ROI around the lanes into a bird's-eye view image to make the lanes parallel and remove distortion.
It uses this image to segment the lane boundaries.

To help improve the algorithm results, first clickSettings. You can change any of these settings.

的placement of the lane lines for generating the bird's-eye view image
的ROI around the lanes, which you can expand to include more than just the ego lane boundaries in the image
的pixel width of detected lane boundaries in the image

You can also change the number of lane boundaries that you want to detect. The default number of lane boundaries is2.

Version History

Introduced in R2018b

Video Labeler

Description

Open the Video Labeler App

Examples

Programmatic Use

`videoLabeler`

`videoLabeler(videoFileName)`

`videoLabeler(imageSeqFolder)`

`videoLabeler(imageSeqFolder,timestamps)`

`videoLabeler(gtSource)`

`videoLabeler(sessionFile)`

`videoLabeler(gTruth)`

Limitations

More About

ROI Labels, Sublabels, and Attributes

Tips

Algorithms

ACF People Detector

Point Tracker

Temporal Interpolator

ACF Vehicle Detector(requiresAutomated Driving Toolbox)

Lane Boundary Detector(requiresAutomated Driving Toolbox)

Version History

See Also

Apps

Functions

Objects

Topics