Next Generation Developer Testing: Parameterized Testing

Next Generation Developer Testing:
Parameterized Testing
Tao Xie
Department of Computer Science
University of Illinois at Urbana-Champaign
Email: taoxie@illinois.edu
http://taoxie.cs.illinois.edu/
In Collaboration with Microsoft Research

2
Introduction
 Developer (aka unit) testing – a widely adopted practice for
ensuring high quality software
 A conventional unit test (CUT): small program with test inputs
and test assertions
void AddTest() {
HashSet set = new HashSet();
set.Add(7);
set.Add(3);
Assert.IsTrue(set.Count == 2);
}
Test Scenario
Test Assertions
Test Data

3
Parameterized Unit Tests (PUTs)
 Recent advances in unit testing introduced PUTs
void AddSpec(int x, int y)
{
HashSet set = new HashSet();
set.Add(x);
set.Add(y);
Assert.AreEqual(x == y, set.Count == 1);
Assert.AreEqual(x != y, set.Count == 2);
}
 PUTs separate two concerns:
• Specification of externally visible behavior (assertions)
• Selection of internally relevant test inputs (coverage)

4
Parameterized Unit Tests (PUTs)
 More beneficial than CUTs
• Help describe behaviors for all test arguments
 Address two main issues with CUTs
• Missing test data required for exercising important
behaviors
 Low fault-detection capability
• Including test data that exercises the same behaviour
 Redundant unit tests

5
An Example using IntStack
public void CUT1() {
int elem = 1;
IntStack stk = new IntStack();
stk.Push(elem);
Assert.AreEqual(1, stk.Count());
}
Three CUTs
int elem = 30;
stk.Push(elem);
}
int elem1 = 1, elem2 = 30;
stk.Push(elem1);
stk.Push(elem2);
}
 CUT1 and CUT2 exercise push with
different test data
 CUT3 exercises push when stack is not
empty
Two main issues with CUTs:
 Fault-detection capability issue:
undetected defect where things go wrong
when passing a negative value to push
 Redundant test issue: CUT2 is
redundant with respect to CUT1

6
An Example using IntStack
int elem = 1;
stk.Push(elem);
}
Three CUTs
int elem = 30;
stk.Push(elem);
}
stk.Push(elem1);
stk.Push(elem2);
}
 No need to describe test data
• Generated automatically
 Single PUT replaces multiple CUTs
• With reduced size of test code
public void PUT(int[] elem) {
Assume.IsTrue(elem != null);
for(int i in elem)
stk.push(elem);
Assert.AreEqual(elem.Length, stk.count());
}
An equivalent PUT

Parameterized Unit Tests are
Algebraic Specifications
• A Parameterized Unit Test can be read as a
universally quantified, conditional axiom.
void ReadWrite(Storage s, string name, string data) {
Assume.IsTrue(s!=null && name!=null && data!=null);
s.WriteResource(name, data);
var readData = s.ReadResource(name);
Assert.AreEqual(data, readData);
}
 Storage s, string name, string data:
s ≠ null ⋀ name ≠ null ⋀ data ≠ null ⇒
equals(
ReadResource(WriteResource(s,name,data).state, name).retval,
data)

Parameterized Unit Testing
is going mainstream
Parameterized Unit Tests (PUTs) commonly supported by various
test frameworks
• .NET: Supported by .NET test frameworks
– http://www.mbunit.com/
– http://www.nunit.org/ …
• Java: Supported by JUnit 4.X
– http://www.junit.org/
Generating test inputs for PUTs supported by tools
• .NET: Supported by Microsoft Visual Studio 2015 IntelliTest
– Formerly Microsoft Research Pex: http://research.microsoft.com/pex/
• Java: Supported by Agitar AgitarOne
– http://www.agitar.com/

Parameterized Tests in JUnit
9
https://github.com/junit-team/junit/wiki/Parameterized-tests

JUnit Theories
10
https://github.com/junit-team/junit/wiki/Theories

Assumptions and Assertions
void PexAssume.IsTrue(bool c) {
if (!c)
throw new AssumptionViolationException();
}
void PexAssert.IsTrue(bool c) {
if (!c)
throw new AssertionViolationException();
}
• Assumptions and assertions induce branches
• Executions which cause assumption violations are
ignored, not reported as errors or test cases

Test Data Generation
• Human
– Expensive, incomplete, …
• Brute Force
– Pairwise, predefined data, etc…
• Semi - Random
– Cheap, Fast
– “It passed a thousand tests” feeling
• Dynamic Symbolic Execution:
IntelliTest/Pex, SAGE, CUTE, …
– Automated white-box
– Not random – Constraint Solving

13
void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
T
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
a==null &&
a.Length>0 &&
a[0]==1234567890
 Generates test data systematically
Done: There is no path left.
Background: DSE

Pex4Fun
http://pex4fun.com/
Nikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith Bishop. Teaching and Learning
Programming and Software Engineering via Interactive Gaming. In ICSE 2013, SEE.
http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf
1,703,247 clicked 'Ask Pex!'

Pex4Fun
• Click http://pex4fun.com/default.aspx?language=CSharp&sa
mple=_Template
• Copy and modify the following code snippet in the code
editing box (or simply click here)
using System;
using Microsoft.Pex.Framework;
using Microsoft.Pex.Framework.Settings;
[PexClass]
public class Program
{
[PexMethod]//[PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)]
public static string testMethod(int x, int y)
{
PexAssume.IsTrue(y >= 0);//replace here with your assumption
//... enter your code under test here
//if (x == 10000) throw new Exception();
PexAssert.IsTrue(y >= 0);//replace here with your assertion
return PexSymbolicValue.GetPathConditionString();
}
}
https://sites.google.com/site/teachpex/Home/pex-usage-tips

Microsoft Visual Studio 2015
IntelliTest
16
https://msdn.microsoft.com/en-us/library/dn823749.aspx

Microsoft Visual Studio 2015
IntelliTest
17https://msdn.microsoft.com/en-us/library/dn823749.aspx

18
Recall: An Example using IntStack
int elem = 1;
stk.Push(elem);
}
Three CUTs
int elem = 30;
stk.Push(elem);
}
stk.Push(elem1);
stk.Push(elem2);
}
 No need to describe test data
• Generated automatically
 Single PUT replaces multiple CUTs
• With reduced size of test code
public void PUT(int[] elem) {
Assume.IsTrue(elem != null);
for(int i in elem)
stk.push(elem);
Assert.AreEqual(elem.Length, stk.count());
}
An equivalent PUT

19
Test Generalization: CUTs  PUT
Major Steps
• S1: Parameterize
• S2: Generalize Test Oracle
• S3: Add Assumptions
Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011
http://taoxie.cs.illinois.edu/publications/fase11-put.pdf

Example
00: public class SettingsGroup{
01: MSS storage; ...
02: public SettingsGroup(MSS storage) {
03: this.storage = storage;
04: }
05: public void SaveSetting(string sn, object sv) {
06: object ov = storage.GetSetting( sn );
07: //Avoid change if there is no real change
08: if (ov != null ) {
09: if(ov is string && sv is string && (string)ov==(string)sv ||
10: ov is int && sv is int && (int)ov==(int)sv ||
11: ov is bool&& sv is bool&& (bool)ov==(bool)sv ||
12: ov is Enum&& sv is Enum&& ov.Equals(sv))
13: return;
14: }
15: storage.SaveSetting(sn, sv);
16: if (Changed != null)
17: Changed(this, new SettingsEventArgs(sn));
18: }}
20
SettingsGroup class of NUnit with
the SaveSetting method under test

An Existing CUT
00: public class SettingsGroup{
01: MSS storage; ...
02: public SettingsGroup(MSS storage) {
03: this.storage = storage;
04: }
05: public void SaveSetting(string sn, object sv) {
08: if (ov != null ) {
09: if(ov is string && sv is string && (string)ov==(string)sv ||
10: ov is int && sv is int && (int)ov==(int)sv ||
11: ov is bool&& sv is bool&& (bool)ov==(bool)sv ||
12: ov is Enum&& sv is Enum&& ov.Equals(sv))
13: return;
14: }
15: storage.SaveSetting(sn, sv);
16: if (Changed != null)
17: Changed(this, new SettingsEventArgs(sn));
18: }}
21
00: //tg is of type SettingsGroup
01: [Test]
02: public void TestSettingsGroup() {
03: tg.SaveSetting("X",5);
04: tg.SaveSetting("NAME","Tom");
05: Assert.AreEqual(5,tg.GetSetting("X"));
06: Assert.AreEqual("Tom",tg.GetSetting("NAME"));
07: }
Existing CUT
SettingsGroup class of NUnit with
the SaveSetting method under test

?
Issues with Existing CUT
22
Only CUT for verifying SaveSetting method
• Does not verify the behavior for the types bool and enum
• Does not cover the true branch in Line 8
Does test generalization addresses these two
issues
…
08: if (ov != null ) { …

23
S1 - Parameterize
 Promote all primitive values as arguments
• name of setting as a parameter of type string
• string “TOM” and int 5 as a parameter of type object
helping IntelliTest/Pex generate concrete values based on the
constraints encountered in different paths
 Promote non-primitive objects such as receiver objects
as arguments
helping IntelliTest/Pex generate object states for the receiver
objects that can cover additional paths
//Original CUT
04: tg.SaveSetting("NAME","Tom");….
//New PUT
02: public void TestSave(
SettingsGroup st,
string sn, object sv){
….

24
S2 – Generalize Test Oracle
 Replace the constant value, “TOM” and 5, with
the relevant parameter of the PUT
//Original CUT
04: tg.SaveSetting("NAME","Tom");
05: Assert.AreEqual(5,tg.GetSetting("X"));
06: Assert.AreEqual("Tom",tg.GetSetting("NAME"));
….
//New PUT
02: public void TestSave(SettingsGroup st,
03: st.SaveSetting(sn, sv);
04: PexAssert.AreEqual(sv,st.GetSetting(sn));

//New PUT
02: public void TestSave(SettingsGroup st,
25
S3 – Add Assumptions
IntelliTest/Pex requires guidance in generating
legal values for the parameters
E.g., Add the tag PexAssumeUnderTest (PAUT) with the parameter, i.e.,
generated value should not be null
//New PUT
02: public void TestSave([PAUT]SettingsGroup
st, [PAUT]string sn, [PAUT] object sv){

26
Example Summary
 Resulting PUT
Result of S1  Parameters “sn” and “sv”
 Result of S2  Line 4: “st.GetSetting(sn)”
 Result of S3  Parameter “st”
00: //PAUT: PexAssumeUnderTest
01: [PexMethod]
02: public void TestSave([PAUT]SettingsGroup st,
[PAUT] string sn, [PAUT] object sv) {
05: }

27
Example - Test Generalization Results
 Achieved 10% branch cov
 Required 2 method calls
with values of type string and
int
2: public void
TestSave([PAUT]SettingsGroup st,
[PAUT]string sn, [PAUT] object sv){
4: PexAssert.AreEqual
(sv,st.GetSetting(sn));
4: tg.SaveSetting("NAME","Tom");….
 Achieved 90% branch cov
 Required only 1 method call,
sufficient to test for all types
(parameter of type object)
Original CUT New PUT

EXAMPLE PATTERNS
FOR PARAMETERIZED UNIT TESTS
http://research.microsoft.com/pex/patterns.pdf

Pattern
4A
• Assume, Arrange, Act, Assert
[PexMethod]
void Add(List target, T value) {
PexAssume.IsNotNull(target); // assume
var count = target.Count; // arrange
target.Add(value); // act
Assert(target.Count == count + 1)//assert
}

Pattern
Roundtrip
• For an API f(x), f-1(f(x)) = x for all x
void ToStringParseRoundtrip(int value) {
string s = value.ToString();
int parsed = int.Parse(s);
Assert.AreEqual(value, parsed);
}

Pattern
State Relation
• Observe a state change
void ContainedAfterAdd(string value) {
var list = new List<string>();
list.Add(value);
Assert.IsTrue(list.Contains(value));
}

Pattern
Commutative Diagram
• Given two implementations f and g of the same
function, each possible requiring a different number
of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(…
(gm(x)…)), then it should hold that
f1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.
string Multiply(string x, string y);
int Multiply(int x, int y);
void CommutativeDiagram1(int x, int y) {
string z1 = Multiply(x, y).ToString();
string z2 = Multiply(x.ToString(), y.ToString());
PexAssert.AreEqual(z1, z2);
}

Code Hunt: Turning Pex/PUT into
a Coding Game
https://www.codehunt.com/

Behind the Scene of Code Hunt
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
34

It’s a game!
• iterative gameplay
• adaptive
• personalized
• no cheating
• clear winning criterion
code
test cases
Bishop et al. Code Hunt: Experience with Coding Contests at Scale. ICSE 2015, JSEET
http://taoxie.cs.illinois.edu/publications/icse15jseet-codehunt.pdf

Next Generation Developer
Testing: Parameterized Testing
Microsoft
Visual Studio 2015
- IntelliTest
JUnit TheoriesPexMethod
Test Generalization
CUTs  PUTs
PUT Patterns
 PUTs
…

Thank You
37
https://sites.google.com/site/teachpex/
http://research.microsoft.com/pex
http://taoxie.cs.illinois.edu/
Collaboration is Welcome!
Email: taoxie@illinois.edu

Next Generation Developer
Testing: Parameterized Testing
Microsoft
Visual Studio 2015
- IntelliTest
JUnit TheoriesPexMethod
Test Generalization
CUTs  PUTs
PUT Patterns
 PUTs
…
http://taoxie.cs.illinois.edu/taoxie@illinois.edu
https://sites.google.com/site/teachpex/

Pattern
Roundtrip
• For an API f(x), f-1(f(x)) = x for all x
void PropertyRoundtrip(string value) {
var target = new Foo();
target.Name = value;
var roundtripped = target.Name;
Assert.AreEqual(value, roundtripped);
}

Pattern
Sanitized Roundtrip
• For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x
void ParseToString(string x) {
var normalized = int.Parse(x);
var intermediate = normalized.ToString();
var roundtripped = int.Parse(intermediate);
Assert(normalized == roundtripped);
}

Pattern
Reachability
• Indicate which portions of a PUT should be
reachable.
[PexExpectedGoals]
public void InvariantAfterParsing(string input)
{
ComplexDataStructure x;
bool success = ComplexDataStructure.TryParse(
input, out x);
PexAssume.IsTrue(success);
PexGoal.Reached();
x.AssertInvariant();
}

Pattern
Regression Tests
• Generated test asserts any observed value
– Return value, out parameters, PexGoal
• When code evolves, breaking changes in
observable will be discovered
int AddTest(int a, int b) {
return a + b; }
void AddTest01() {
var result = AddTest(0, 0);
Assert.AreEqual(0, result);
} http://research.microsoft.com/pex/patterns.pdf

Pattern
Same Observable Behavior
• Given two methods f(x) and g(x), and a method b(y)
that observes the result or the exception behavior of
a method, assert that f(x) and g(x) have same
observable behavior under b, i.e. b(f(x)) = b(g(x))
for all x.
public void ConcatsBehaveTheSame(
string left, string right)
{
PexAssert.AreBehaviorsEqual(
() => StringFormatter.ConcatV1(left, right),
() => StringFormatter.ConcatV2(left, right));
}

Pattern
Allowed Exception
• Allowed exception -> negative test case
[PexAllowedException(typeof(ArgumentException))]
void Test(object item) {
var foo = new Foo(item) // validates item
// generated test (C#)
[ExpectedException(typeof(ArgumentException))]
void Test01() {
Test(null); // argument check
}

45
S4 – Add Factory Method
 IntelliTest/Pex requires guidance handling non-
primitive objects
e.g., Add factory methods that generate instances of non-primitive objects
00: //PAUT: PexAssumeUnderTest
01: //MSS: MemorySettingsStorage (class)
01: [PexFactoryMethod(typeof(MSS))]
02: public static MSS Create([PAUT] string[] sn, [PAUT]object[] sv) {
03: PexAssume.IsTrue(sn.Length == sv.Length);
04: PexAssume.IsTrue(sn.Length > 0);
05: MSS mss = new MSS();
06: for(int count = 0; count < sn.Length; count++) {
07: mss.SaveSetting(sn[count], sv[count]);
08: }
09: return mss;
10: }
• Help create different object states for MSS

46
S5 – Add Mock Object
 Pex faces challenges in handling code that
interacts with external environment, e.g., file system
• Write mock objects to assist Pex and to test features in isolation

47
Empirical Study
 RQ1: Branch coverage
• How much higher percentage of branch coverage is achieved
by retrofitted PUTs compared to existing CUTs?
 RQ2: Defect detection
• How many new defects (that are not detected by existing
CUTs) are detected by PUTs and vice-versa?
 RQ3: Generalization effort
• How much effort is required for generalizing CUTs to PUTs?

Study Setup
 Three Subject Applications
48
Subjects Downloads Code Under Test
# Classes # Methods KLOC
Test Code
# Classes # CUTs KLOC
NUnit 193,563 9 87 1.4 9 49 0.9
DSA 3,239 27 259 2.4 20 337 2.5
QuickGraph 7,969 56 463 6.2 9 21 1.2
TOTAL 92 809 10 38 407 4.6
 High downloads count
 Lines of code under test: 10 KLOC
 Number of CUTs: 407

Study Setup – cont.
49
 Conducted by the first and second authors
• No prior knowledge of subjects
• Two years of experience with PUTs and Pex
• Retrofitted 407 CUTs (4.6 KLOC) as 224 PUTs (4.0 KLOC)
Generated three categories of CUTs
• C1: Existing CUTs
• C2: CUTs generated from PUTs
• C3: Existing CUTs + RTs (tests generated using Randoop)
 Help show that benefits of generalization cannot be achieved by
generating tests randomly

RQ1: Branch Coverage
50
Subjects # RTs Branch Coverage
CUTs (%) CUTs + RTs (%) PUTs (%)
Overall
Increase
(%)
Maximum
Increase
(%)
NUnit 144 78 78 88 10 52
DSA 615 91 91 92 1 1
QuickGrap
h
3628 87 88 89 2 11
CUTs + RTs:
 Added 144, 615, and 3628 tests using Randoop
 Branch coverage increased by 0%, 0%, and 1%
CUTs generated from PUTs:
 Branch coverage increased by 10%, 1%, and 2%

RQ2: Defect Detection
51
 RTs: CUTs generated using Randoop
 Pex without generalized PUTs: CUTs generated by applying Pex on public
methods
 CUTs generated from PUTs: CUTs generated by applying Pex on generalized PUTs
• Detected all defects detected by the first two categories
CUTs category #Failing Tests
DSA NUnit QuickGraph
# Real Defects
Basic CUTs 0 0 0 0
RTs 90 25 738 4
Pex without
generalized PUTs
23 170 17 2
CUTs generated
from PUTs
15 4 0 19

RQ3: Generalization Effort
52
 Both authors conducted comparable amount of
generalization
 Effort spent in hours
• NUnit: 2.8 hrs
• DSA: 13.8 hrs
• QuickGraph: 1.5 hrs
 Effort is worthwhile compared to benefits of test
generalization

Next Generation Developer Testing: Parameterized Testing

More Related Content

What's hot

Similar to Next Generation Developer Testing: Parameterized Testing

More from Tao Xie

Recently uploaded

Next Generation Developer Testing: Parameterized Testing