Ajay Puvvala is my friend and colleague at the MathWorks. He also led the development of the TestTags feature we spoke aboutlast post.

Ajay, as per usual, made a great comment after last week's post which I wanted to highlight here. It had to do with the test pyramid:

You know thetest pyramid(another)? I am sure we will have more posts discussing it in the future, but until then the fundamental principle is that you want to form a solid foundation of unit tests followed by fewer integration tests and even fewer system tests. Manual regression tests (not to be confused with exploratory and/or user testing) should be kept to an absolute minimum, although we shouldn't ignore them when needed. The rationale here is that the farther you go up in the testing pyramid the more expensive these tests become. They are harder to write, to maintain, to run, and to make robust. They are needed, but we should opt for the pyramid as opposed to theice cream cone.

Anyway, Ajay mentioned that a test suite using tags such as the Small/Medium/Large tags discussed last post can help us get a sense of the health of our pyramid or ice cream cone. Yes I am a realist, I know that many of us have cones instead of pyramids. How do we get a sense of where we are?

To see this I will create a few tests that leveragetest parameterizationto quickly create a large test array.

classdefPyramidSuite < matlab.unittest.TestCase% This test class contains a nice pyramid distribution of unit tests to% system tests.properties(TestParameter)% To create a representative suite, create 3 parameters and implement test% methods and use different subsets of these parameters in the test% methods. Since they are combined exhaustively, the more parameters a test% uses the more test elements will get created.param1 = num2cell(1:20); param2 = num2cell(1:5); param3 = num2cell(1:3);endmethods(Test, TestTags={'Small'})functionsmallTest(testCase, param1, param2, param3)endendmethods(Test, TestTags={'Medium'})functionmediumTest(testCase, param1, param2)endendmethods(Test, TestTags={'Large'})functionlargeTest(testCase, param1)endendend

Here I have created a nicely balanced test suite with a healthy use of small tests and lower use of larger tests in the suite. How can I be sure of this? By looking at the test tags.

importmatlab.unittest.TestSuite;pyramidSuite = TestSuite.fromClass(?PyramidSuite); pyramidSizes = categorical([pyramidSuite.Tags]); pie(pyramidSizes, [1 1 1]);

Look at that! Super simple to get a high level view of your test bed. Note this does assume that each test is tagged with one and only one of the Small/Medium/Large tags we are operating with here.

However this doesn't look like a pyramid, it looks like a pie! Alright, just for fun let's make our own test pyramid with our test tag data. We'll start by getting the tag counts, sorting them so we can build our pyramid correctly, and assign colors to reflect our preference for smaller, less expensive tests.

[counts, sizes] = histcounts(pyramidSizes);% Sort the values to allow creation of the pyramid[counts, index] = sort(counts); sizes = sizes(index);% Assign:% small tests: green% medium tests: yellow% large tests: redcolors{strcmp(sizes,'Small')} ='g';colors{strcmp(sizes,'Medium')} ='y';colors{strcmp(sizes,'Large')} ='r';

Great, now we can calculate a few values for the sizes of our triangles required to create a stacked pyramid where the area of each stack matches the percentage of the test suite matching the category. I'll leave the calculation here as an exercise for the reader, but the constraints at play are that the area of the largest containing triangle is 1, it's base and height are both chosen to be $\sqrt{2}$, and the areas of each section match the percentage of corresponding categories in the suite.

total = sum(counts); percentMiddle = counts(2)/total; percentTop = counts(1)/total; heightBottom = sqrt(2); heightMiddle = sqrt(2*percentMiddle+percentTop); heightTop = sqrt(2*percentTop); baseBottom = sqrt(2); baseMiddle = (heightMiddle/heightBottom)*baseBottom; baseTop = (heightTop/heightBottom)*baseBottom;

Now create the endpoints of each trapezoid and the top triangle using these dimensions. Place the endpoints into an Nx2 array corresponding to(x,y)coordinates.

%Bottom sectionbottom(1,:) = [0 0]; bottom(2,:) = [(baseBottom-baseMiddle)/2, heightBottom-heightMiddle]; bottom(3,:) = [bottom(2,1) + baseMiddle, bottom(2,2)]; bottom(4,:) = [baseBottom, 0];% Middle sectionmiddle(1,:) = bottom(2,:); middle(2,:) = [middle(1,1) + (baseMiddle-baseTop)/2, heightBottom-heightTop]; middle(3,:) = [middle(2,1) + baseTop, middle(2,2)]; middle(4,:) = bottom(3,:);% Top sectiontop(1,:) = middle(2,:); top(2,:) = [baseBottom/2, heightBottom]; top(3,:) = middle(3,:);

...and create the pyramid usingfilland add labels at the centroid of each region.

fill(bottom(:,1), bottom(:,2), colors{3},...middle(:,1), middle(:,2), colors{2},...top(:,1), top(:,2), colors{1}); axisoff;findCentroid = @(points) sum(points,1)/length(points); topCentroid = findCentroid(top); text(topCentroid(1), topCentroid(2), sizes{1},...'HorizontalAlignment','center','FontSize', 14,'FontWeight','bold'); middleCentroid = findCentroid(middle); text(middleCentroid(1), middleCentroid(2), sizes{2},...'HorizontalAlignment','center','FontSize', 14,'FontWeight','bold'); bottomCentroid = findCentroid(bottom); text(bottomCentroid(1), bottomCentroid(2), sizes{3},...'HorizontalAlignment','center','FontSize', 14,'FontWeight','bold');

There ya have it, a true testing pyramid built from the data found in the test tags. To convince ourselves this is working lets put it in a function and try it out on a different suite. Here is the function:

function处理= testPyramid (testSizes)(数量、尺寸)=histcounts(testSizes);% Sort the values to allow creation of the pyramid[counts, index] = sort(counts); sizes = sizes(index);% Assign:% small tests: green% medium tests: yellow% large tests: redcolors{strcmp(sizes,'Small')} ='g';colors{strcmp(sizes,'Medium')} ='y';colors{strcmp(sizes,'Large')} ='r';total = sum(counts); percentMiddle = counts(2)/total; percentTop = counts(1)/total; heightBottom = sqrt(2); heightMiddle = sqrt(2*percentMiddle+percentTop); heightTop = sqrt(2*percentTop); baseBottom = sqrt(2); baseMiddle = (heightMiddle/heightBottom)*baseBottom; baseTop = (heightTop/heightBottom)*baseBottom;% Bottom sectionbottom(1,:) = [0 0]; bottom(2,:) = [(baseBottom-baseMiddle)/2, heightBottom-heightMiddle]; bottom(3,:) = [bottom(2,1) + baseMiddle, bottom(2,2)]; bottom(4,:) = [baseBottom, 0];% Middle sectionmiddle(1,:) = bottom(2,:); middle(2,:) = [middle(1,1) + (baseMiddle-baseTop)/2, heightBottom-heightTop]; middle(3,:) = [middle(2,1) + baseTop, middle(2,2)]; middle(4,:) = bottom(3,:);% Top sectiontop(1,:) = middle(2,:); top(2,:) = [baseBottom/2, heightBottom]; top(3,:) = middle(3,:); handles = fill(bottom(:,1), bottom(:,2), colors{3},...middle(:,1), middle(:,2), colors{2},...top(:,1), top(:,2), colors{1}); axisoff;处理结束(+ 1)= labelSections(顶部,大小{1});handles(end+1) = labelSections(middle, sizes{2}); handles(end+1) = labelSections(bottom, sizes{3});functiontextHandle = labelSections(points, label) centroid = sum(points,1)/length(points); textHandle = text(centroid(1), centroid(2), label,...'HorizontalAlignment','center','FontSize', 14,'FontWeight','bold');

Create the suite using the same parameterized testing tactic, but this time try it on a poorly implemented suite that relies too much on'Large'tests. I want to build the ice cream cone:

classdefIceCreamConeSuite < matlab.unittest.TestCase% This test class clearly relies on a bit too many large tests and not% enough small tests.properties(TestParameter)% Tweak the distribution of our parameters a bit from% the PyramidSuite to create different percentagesparam1 = num2cell(1:20); param2 = num2cell(1:2); param3 = num2cell(1:2);endmethods(Test, TestTags={'Small'})functionsmallTest(testCase, param1)endendmethods(Test, TestTags={'Medium'})functionmediumTest(testCase, param1, param2)endendmethods(Test, TestTags={'Large'})functionlargeTest(testCase, param1, param2, param3)endendend

iceCreamConeSuite = TestSuite.fromClass(?IceCreamConeSuite); iceCreamConeSizes = categorical([iceCreamConeSuite.Tags]); handles = testPyramid(iceCreamConeSizes);

Looks correct, we now can simply rotate it to create the ice cream cone.

rotate(handles, [0 0 1], 180);

Do you strive for the test pyramid? Have you found any other methods for introspecting into your test bed to see how you are doing?

Get the MATLAB code

Published with MATLAB® R2015b