Overview of Performance Testing Framework

The performance test interface leverages the script, function, and class-based unit testing interfaces. You can perform qualifications within your performance tests to ensure correct functional behavior while measuring code performance. Also, you can run your performance tests as standard regression tests to ensure that code changes do not break performance tests.

Determine Bounds of Measured Code

This table indicates what code is measured for the different types of tests.

Type of Test	什么是测量	What Is Excluded
Script-based	代码我n each section of the script	代码我n the shared variables section Measured estimate of the framework overhead
Function-based	代码我n each test function	代码我n the following functions:`setup`,`setupOnce`,`teardown`, and`teardownOnce` Measured estimate of the framework overhead
Class-based	代码我n each method tagged with the`Test`attribute	代码我n the methods with the following attributes:`TestMethodSetup`,`TestMethodTeardown`,`TestClassSetup`, and`TestClassTeardown` Shared fixture setup and teardown Measured estimate of the framework overhead
Class-based deriving from`matlab.perftest.TestCase`and using`startMeasuring`and`stopMeasuring`methods	Code between calls to`startMeasuring`and`stopMeasuring`in each method tagged with the`Test`attribute	Code outside of the`startMeasuring`/`stopMeasuring`boundary Measured estimate of the framework overhead
Class-based deriving from`matlab.perftest.TestCase`and using the`keepMeasuring`method	代码我nside each`keepMeasuring-while`loop in each method tagged with the`Test`attribute	Code outside of the`keepMeasuring-while`boundary Measured estimate of the framework overhead

时间类型的实验

You can create two types of time experiments.

Afrequentist time experimentcollects a variable number of measurements to achieve a specified margin of error and confidence level. Use a frequentist time experiment to define statistical objectives for your measurement samples. Generate this experiment using therunperffunction or thelimitingSamplingErrorstatic method of theTimeExperimentclass.
Afixed time experimentcollects a fixed number of measurements. Use a fixed time experiment to measure first-time costs of your code or to take explicit control of your sample size. Generate this experiment using thewithFixedSampleSizestatic method of theTimeExperimentclass.

This table summarizes the differences between the frequentist and fixed time experiments.

	Frequentist time experiment	Fixed time experiment
Warm-up measurements	4 by default, but configurable through`TimeExperiment.limitingSamplingError`	0 by default, but configurable through`TimeExperiment.withFixedSampleSize`
Number of samples	Between 4 and 256 by default, but configurable through`TimeExperiment.limitingSamplingError`	Defined during experiment construction
Relative margin of error	5% by default, but configurable through`TimeExperiment.limitingSamplingError`	Not applicable
Confidence level	95% by default, but configurable through`TimeExperiment.limitingSamplingError`	Not applicable
联邦铁路局mework behavior for invalid test result	Stops measuring a test and moves to the next one	Collects specified number of samples

Write Performance Tests with Measurement Boundaries

If your class-based tests derive frommatlab.perftest.TestCaseinstead ofmatlab.unittest.TestCase, then you can use thestartMeasuringandstopMeasuringmethods or thekeepMeasuringmethod multiple times to define boundaries for performance test measurements. If a test method has multiple calls tostartMeasuring,stopMeasuringandkeepMeasuring, then the performance testing framework accumulates and sums the measurements. The performance testing framework does not support nested measurement boundaries. If you use these methods incorrectly in aTestmethod and run the test as aTimeExperiment, then the framework marks the measurement as invalid. Also, you still can run these performance tests as unit tests. For more information, seeTest Performance Using Classes.

Run Performance Tests

There are two ways to run performance tests:

Use therunperffunction to run the tests. This function uses a variable number of measurements to reach a sample mean with a 0.05 relative margin of error within a 0.95 confidence level. It runs the tests four times to warm up the code and between 4 and 256 times to collect measurements that meet the statistical objectives.
Generate an explicit test suite using thetestsuitefunction or the methods in theTestSuiteclass, and then create and run a time experiment.
- Use thewithFixedSampleSizemethod of theTimeExperimentclass to construct a time experiment with a fixed number of measurements. You can specify a fixed number of warm-up measurements and a fixed number of samples.
- Use thelimitingSamplingErrormethod of theTimeExperimentclass to construct a time experiment with specified statistical objectives, such as margin of error and confidence level. Also, you can specify the number of warm-up measurements and the minimum and maximum number of samples.

You can run your performance tests as regression tests. For more information, seeTest Performance Using Classes.

Understand Invalid Test Results

In some situations, theMeasurementResultfor a test result is marked invalid. A test result is marked invalid when the performance testing framework sets theValidproperty of theMeasurementResultto false. This invalidation occurs if your test fails or is filtered. Also, if your test incorrectly uses thestartMeasuringandstopMeasuringmethods ofmatlab.perftest.TestCase, then theMeasurementResultfor that test is marked invalid.

When the performance testing framework encounters an invalid test result, it behaves differently depending on the type of time experiment:

If you create a frequentist time experiment, then the framework stops measuring for that test and moves to the next test.
If you create a fixed time experiment, then the framework continues collecting the specified number of samples.