Continuous Integration for Verification of Simulink Models

This is the first article in a two-part series. Part one looks at leveraging GitLab^®for version control and Jenkins for continuous integration (CI). Part 2,Continuous Integration for Verification of Simulink Models Using GitLab, looks at using GitLab for both version control and CI.

Continuous integration (CI) is gaining in popularity and becoming an integral part of Model-Based Design. But what is CI? What are its benefits, and what problems does it attempt to solve? How does Simulink^®fit into the CI ecosystem? And how can you best leverage CI for your projects?

If you are familiar with Model-Based Design but new to CI, you may be asking yourself these questions. In this technical article, we explore a common CI workflow and apply it to Model-Based Design. Then, we walk through an example of that workflow using Jenkins™, GitLab^®, and Simulink Test™.

The project used in this example isavailable for download.

What Is CI?

CI is an agile methodology best practice in which developers regularly submit and merge their source code changes into a central repository. These “change sets” are then automatically built, qualified, and released. Figure 1 illustrates this basic CI workflow together with the development workflow.

In the development part of the workflow, models and tests are developed, verified, merged, reviewed, and submitted to a version control system on the developer desktop. The version control system then triggers the automated CI portion of the workflow. The key parts of the CI workflow are:

Build:Source code and models become object files and executables.

Test:Testing is performed as a quality gate.

Package:Executables, documentation, artifacts, and other deliverables are bundled for delivery to end users.

Deploy:Packages are deployed to the production environment.

Together, these four steps are known as the CI “pipeline.” The pipeline is typically automated, and it can take anywhere from minutes to days to complete, depending on the system. It is worth noting that throughout these steps, numerous artifacts are created, such as bills of materials, test results, and reports.

CI workflows are often paired with developer workflows related to version control systems. In these workflows, developers often keep their changes in local repositories and use a local CI pipeline to qualify their changes prior to deployment.

What Are the Benefits of CI?

Teams that have implemented CI typically report the following benefits:

• Repeatability.The CI pipeline provides a consistent and repeatable automated process for building, testing, packaging, and deployment. Repeatable automation allows developers to focus on necessary work and save time on a project. It is also an important aspect of risk reduction, and is often a requirement for certification.
• Quality assurance.Manual testing is effective, but it is often based on days-old snapshots and lacks repeatability. With CI, changes are always tested against the most recent code base.
• Reduced development time.Repeatable processes with built-in quality assurance lead to faster delivery of high-quality products. Automated deployment means that your code is always production-ready.
• Improved collaboration.With CI, developers have a defined process for managing change sets and merging their code into the production line. Consistent processes make managing large teams possible and reduce the cost of ramping up new developers.
• Audit-ready code.The CI workflow provides an extensive audit trail. For every change making its way through the CI pipeline, it’s possible to identify who made the change, who reviewed it, and the nature of the change, as well as dependencies, tests and their results, and any number of related reports and artifacts generated along the way.

How Does Model-Based Design Fit into CI?

By design, the CI workflow and tools are language- and domain-neutral. This means that the challenge is toteachCI tools, systems, and processes tospeakModel-Based Design—in other words, to make Simulink^®and related tools thelingua francaof the CI workflow.

This can be done by integrating three key components of Model-Based Design into the CI workflow: verification, code generation, and testing (Figure 2). Model-Based Design emphasizes early verification, which maps to the CI pipeline with a Verify phase before the Build phase. Code generation takes place in the Build phase. Dynamic testing through simulation and static analysis of generated code can be done in the Test phase.

Here is an overview of how weteachthe CI workflow tospeakModel-Based Design:

Develop.MATLAB^®, Simulink, coders, and toolboxes are used for development activities. MATLAB Projects are used to organize work, collaborate, and interface with version control systems.

测试。Simulink Check™ is used to perform model quality checks before simulation and code generation. Simulink Test™ is used to develop, manage, and execute simulation-based tests. Simulink Coverage™ is used to measure coverage and assess test effectiveness. The quality checks, test results, and coverage metrics can then be used as a quality gate for developers to qualify their work.

Merge.The Compare Files and Folders feature of MATLAB is used to compare and merge MATLAB files. The Model Comparison tool is used to compare and merge Simulink models.

Review.审查质量过程的最后一步fore changes are submitted to the version control system. Changes to MATLAB scripts and Simulink models are reviewed here. Test results from prequalification are also reviewed as a final quality gate prior to submission.

Submit.MATLAB Projects provide an interface to version control systems.

Verify.Simulink Check, the same tool used for local verification, is used for automated verification within the CI system.

Build.MATLAB Coder™, Simulink Coder™, and Embedder Coder^®are used to generate code for software-in-the-loop (SIL) testing.

测试。Simulink Test, the same tool used for local testing, is used for automated testing within the CI system.

Package and deploy.Packaging is where executables, documentation, artifacts, and other deliverables are bundled up for delivery to end users. Deployment is the release of the packaged software. In workflows for Model-Based Design, these phases vary widely among organizations and groups, and often involve bundling different builds and certification artifacts into a product ready for delivery to other teams.

Modern development tools and practices enable developers to create more robust systems and test functionality early and often. When a CI system is integrated into the workflow, unit-level testing and system-level testing are automated. This means that the developer can focus on developing new features, not on verifying that features have been integrated correctly.

The following case study describes a workflow that incorporates CI and Model-Based Design.

Case Study: A Simulink Model Verified, Built, and Tested Within a CI System

In this example we use Model-Based Design with CI to perform requirements-based testing on an automotive lane-following system (Figure 3).

The pipeline we’ll be using (Figure 4) is executed with each Jenkins build.

The phases in the pipeline are as follows:

1. Verify standards compliance: A MATLAB Unit script runs a simple Model Advisor check. The assessment criteria ensure that the model does not have unconnected lines.
2. Build model: A MATLAB Unit test file builds production SIL code for our model. The assessment criteria passes if the build succeeds without warning.
3. Execute test cases: A test suite in Simulink Test uses several driving scenarios to test the lane-following controller. Three assessment criteria are used to verify satisfactory operation of the controller:
   • Collision avoidance: The ego car does not collide with the lead car at any point during the driving scenario.
   • Safe distance maintenance: The time gap between the ego car and the lead car is above 1.5 seconds. The time gap between the two cars is defined as the ratio of the calculated headway and the ego car velocity.
   • Lane following: The lateral deviation from the centerline of the lane is within 0.2 m.
4. Package artifacts: Each of the previous stages produces artifacts, including a Model Advisor report, a generated executable, and a set of test results that can be archived for future use or reference.

Workflow Steps

The workflow consists of the following steps (Figure 5):

1. Triggera build in Jenkins and observe that the Verify and Build stages pass.
2. Detecta test case failure in Jenkins.
3. Reproducethe issue on our desktop MATLAB.
4. 修复这个问题in the model by relaxing the assessment criteria.
5. Test locallyto ensure the test case passes.
6. Merge and reviewthe changes on the testing branch.
7. Committhe change to Git and trigger a build in Jenkins.
8. Verify, build, and testin Jenkins.

Our first failed pass through the CI loop is illustrated at the top left. It shows the CI test failure, local reproduction, criteria relaxing, and successful completion of the CI workflow.

Workflow Details

1. We begin bytriggeringa build in Jenkins by selectingBuild Now. The Simulink Check checks and code generation pass.

2. Next, wedetecta test case failure in the second verify phase. The test caseLFACC_Curve_CutInOut_TooClosein test suiteLaneFollowingTestScenariosfails the assessment criteria.

3. To understand the failure better, wereproducethe failure locally using Simulink Test. We open test fileLaneFollowingTestScenarios.mldatxand run test caseLFACC_Curve_CutInOut_TooClose. Note that it fails the Safe Distance assessment criteria. More flexibility in establishing the time gap between the lead car and ego car is required.

4. With an understanding of the problem, we nowfix the issue. We open theLaneFollowingTestBenchExample.slxmodel and navigate to the Collision Detection/Test Assessments Test Sequence block. The first assessment asserts that the time gap between the ego and the lead car should not dip below 1.5 seconds for more than two seconds at a time.

This assessment is too restrictive for the aggressive driving maneuver being tested. For the purposes of this example, we relax the assessment criteria to ensure that the time gap does not dip below 0.8 seconds for more than five seconds at a time.

5. The issue appears fixed in our simulation. To confirm, wetest locally,saving the model and rerunning the test in the test manager. Notice that it passes with the new assessment criteria.

6. We have fixed the issue and verified locally. We now use the Model Comparison tool toreviewthe changes before committing them to version control.

We could also use the Publish feature of the Model Comparison tool to review the code.

7. With the bug fixed, we push these changes to GitLab with MATLAB projects, adding a commit message to note the change to the assessment criteria.

We then note the latest commit in GitLab.

GitLab automatically triggers a build in Jenkins. The Jenkins Project dashboard shows the build status and progress.

8. The Jenkins build runs. We see that theverify, build, and testpipeline phases now pass.

We can now start a merge request to merge the changes in the test branch into the master branch. In GitLab, under Repository we select Branches then clickMerge request下一个最新提交测试分支。

We complete the form and submit the merge request.

As the owners of the branch, we can accept the merge request by clicking the Merge button. All changes are now captured on the main branch.

Using the Example: Tools, Resources, and Requirements

The following sections outline resources to help you get started, the tools you will require, and how they should be configured.

Configuring Systems

Jenkins is leveraged as our CI system and GitLab as our version control system. MATLAB, Jenkins, and GitLab must be configured to work together. The following tutorials will help with setup.

Configure Our MATLAB Project

Configure Jenkins

Configure GitLab to Trigger Jenkins

The tutorials are specific to GitLab and Jenkins, but the concepts can apply to other version control and CI systems.

Tools Required

The following tools are required for this example:

• Jenkinsinstallation version 2.7.3 or later. Jenkins is used for continuous integration.
• MATLAB Plugin for Jenkinsversion 1.0.3 or later. MATLAB, Simulink, and Simulink Test all leverage this plugin to communicate with Jenkins.Learn more on GitHub.
• Additional plugins required:
  • GitLab
  • TAP Plugin
• GitLab亚柯unt. GitLab is used for source control and is available as a cloud service. MATLAB Projects include a Git interface for communication with GitLab.

License Considerations for CI

If you plan to perform CI on many hosts or on the cloud, contactcontinuous-integration@mathworks.comfor help.Note: Transformational products such as MathWorks coder and compiler products may require Client Access Licenses (CAL).

Appendix: Configuring MATLAB, GitLab, and Jenkins

Step 1. Configure MATLAB project to use source control

The first step in our example is to configure our project to use source control with GitLab.

1. Create a new directory namedMBDExampleWithGitAndJenkins, load the example into it, and open the MATLAB ProjectMBDExampleWithGitAndJenkins.prj.
2. In GitLab, create a new project that will be the remote repository. Name itMBDExampleWithGitAndJenkinsand record the URL where it is hosted.
3. In MATLAB, convert the project to use source control. On the Project tab, clickUse Source Control.

ClickAdd Project to Source Control.

4. ClickConvert.

5. ClickOpen Project当完成。

The project is now under local Git source control.

Step 2. Commit changes and push local repository to GitLab

1. On the Project tab, clickRemote.

2. Specify the URL of the remote origin in GitLab.

ClickValidateto ensure the connection to the remote repository is successful and clickOK. The project is now configured to push and pull changes with GitLab.

3. ClickCommitto perform an initial commit.

4. ClickPushto push all changes from the local repository to the remote GitLab repository.

5. Refresh the GitLab dashboard and observe the contents of the MATLAB project.

Step 3: Create testing branch

In this step we create a testing branch for testing and verifying changes before merging with the main branch.

1. ClickBranches.

2. Expand theBranch and Tag Creationsection, name the branch “Test,” and clickCreate.

3. Observe Test in the branch browser. From the Test branch clickSwitchthenClose.

4. In MATLAB selectPushto push these changes to GitLab and observe the Test branch in GitLab.

Step 4: Configure Jenkins to call MATLAB

1. Install two required plugins:
   • GitLab plugin– This plugin allows GitLab to trigger Jenkins builds and display their results in the GitLab UI.
   • MATLAB plugin– This plugin integrates MATLAB with Jenkins and provides the Jenkins interface to call MATLAB and Simulink.
2. SelectNew Itemand create a new FreeStyle project namedMBDExampleUsingGitAndJenkins.
3. Under Source Code Management, enable Git, point Jenkins to our GitLab repository, and enter theTestbranch to build.Note: Login or password and GitLab API token are required.

4. Configure the build trigger to run a build when a push request is made to the Test branch in GitLab. In the Build Triggers section selectAdvanced >secret token. This token is used by GitLab to request a build and authenticate with Jenkins. Make a note of the secret token and the GitLab webhook.

5. Configure the build environment. SelectUse MATLAB versionand enter the MATLAB root.

6. Configure the build step.

ClickAdd build stepand chooseRun MATLAB Command. Enter the commandopenProject('SltestLaneFollowingExample.prj'); LaneFollowingExecModelAdvisor
打开项目和运行模式el advisor checks.

ClickAdd build stepand chooseRun MATLAB Commandagain. Enter the command:openProject('SltestLaneFollowingExample.prj'); LaneFollowingExecControllerBuild

ClickAdd build stepand chooseRun MATLAB Tests. SelectTAP test resultsandCobertura code coverageto complete the build configuration.

Step 5. Publishing TAP results

ClickAdd post-build action > Publish TAP Results. Enter the relative path where the TAP test results will be published.

This action parses the TAP test results and makes them viewable when TAP Extended Test Results is selected. The output contains an overview of the executed test cases, the results summary, and logs from the MATLAB console.

The TAP plugin also collects the results of the latest test executions and displays a health diagram as shown below. You can access any previous build by clicking on the diagram.

Step 6. Publishing HTML reports

ClickAdd post-build action >Publish HTML reports. Enter the relative root path where the HTML report will be published and the filename of the index page that is in the path.

Add as many entries as there are HTML reports to be published. In this scenario, there are two web reports: The Model Advisor Summary and the Code Generation Report. These are standard reports created with MATLAB built-in functions. You can add custom HTML reports.

哟u will find a report link corresponding to the last build for every HTML report on the main Jenkins job page. If you activate the checkbox “Always link to last build” under publishing options, the plugin will publish reports for the last build regardless of the build status. If this checkbox is not activated, the plugin will only link to the last “successful” build.

Step 7. Configuring GitLab to trigger a build in Jenkins

Configure GitLab to trigger an automatic build in Jenkins when a new push occurs on the master branch. To do this, navigate toSettings >人则. Use the webhook URL and the secret token provided by Jenkins in the Build Trigger configuration and selectPush events.

注意:使用完全限定域名在URL中section in place of localhost so that GitLab can find the Jenkins installation.

In theTestpulldown, selectPush Eventsto test the integration. GitLab will show the message “Hook executed successfully: HTTP 200” and Jenkins will start a build.

Step 8. Configuring Jenkins-to-GitLab authentication

To publish a Jenkins build status automatically on GitLab, you must configure the Jenkins-to-GitLab authentication.

1. Create a personal access token on GitLab with API scope selected.

2. Copy the token and create a GitLab connection under Jenkins Configure System.
   Note: Connections can be reused over multiple Jenkins jobs and may be configured globally if the user has at least “maintainer” rights.

Step 9. Integrating Jenkins into the GitLab pipeline

To integrate Jenkins into the GitLab pipeline you must configure the GitLab connection in Jenkins and publish the job status to GitLab.

1. Select the GitLab Connection in the General section of your Jenkins job.

2. Add a post-build action to publish build status to GitLab.
   Note: This action has no parameter and will use the existing GitLab connection to publish the build status on GitLab and create bidirectional traceability for every commit and merge request.

Step 10: Visualizing requirements-based testing metrics (R2020b)

Requirements-basedtesting metricslet you assess the status and quality of your requirements-based testing activities. The metrics results can be visualized using the Model Testing Dashboard.

1. Create a file calledcollectModelTestingResults.mbased on the function shown below. This function will initialize the metrics engine infrastructure and collect all available model metrics.

function collectModelTestingResults() % metric capability added in R2020a if exist('metric') metricIDs = [... "ConditionCoverageBreakdown" "CoverageDataService"... "DecisionCoverageBreakdown" "ExecutionCoverageBreakdown"... "MCDCCoverageBreakdown" "OverallConditionCoverage"... "OverallDecisionCoverage" "OverallExecutionCoverage"... "OverallMCDCCoverage" "RequirementWithTestCase"... "RequirementWithTestCaseDistribution" "RequirementWithTestCasePercentage"... "RequirementsPerTestCase" "RequirementsPerTestCaseDistribution"... "TestCaseStatus" "TestCaseStatusDistribution"... "TestCaseStatusPercentage" "TestCaseTag"... "TestCaseTagDistribution" "TestCaseType"... "TestCaseTypeDistribution" "TestCaseWithRequirement"... "TestCaseWithRequirementDistribution" "TestCaseWithRequirementPercentage"... "TestCasesPerRequirement" "TestCasesPerRequirementDistribution"... ]; % collect all metrics for initial reconcile E = metric.Engine(); execute(E, metricIDs); end end

2. Add this file to your project and to the path.
3. Configure Jenkins to collect metric results by calling the newcollectModelTestingResults函数两次。第一次调用初始化rics integration with Simulink Test Manager. The second collects the metrics results using the exported Simulink Test Manager results.
   1. ClickAdd build stepand chooseRun MATLAB Commandagain. Enter the command:openProject('SltestLaneFollowingExample.prj'); collectModelTestingResults
      Position this build step before theRun MATLAB Testsbuild step.
   2. ClickAdd build stepand chooseRun MATLAB Commandagain. Enter the command again:openProject('SltestLaneFollowingExample.prj'); collectModelTestingResults
      Position this build step after theRun MATLAB Testsbuild step.

iii. CheckSimulink Test Manager resultsin theRun MATLAB Testsbuild step.

4. Archive the metrics results in the derived directory. You must also archive the exported test manager results, as they will allow full navigation of the Metric results when loaded back into MATLAB.

ClickAdd post-build actionand chooseArchive the artifacts.Enter pathderived/**,matlabTestArtifacts/*.mldatxto archive all files saved to that directory.

Note: To view these results in MATLAB on a machine other than the test machine, do the following:

• Download the archived artifacts (derived directory and test results .mldatx files).
• Extract and copy into a local copy of the same version of the project that was used to run the CI job.
• Open the project in MATLAB and launch the Model Testing Dashboard.

The results generated by the CI will be displayed on thedashboard.