eqc_doc.md

Erlang Quickcheck unofficial documentation

This is my attempt to write a documentation to make Erlang Quickcheck a little bit less difficult to grasp. This is an unofficial documentation, so bear in mind that the most reliable source is still the Quiviq official website.

Generators

In Quickcheck, generators are used to generate random test data for our properties. Generators can be combined to form more complex generators with the help of some useful macros defined in the file eqc.hrl.

The simplest form of generator is a constant, which can be used to generate its own value (e.g. 5, foo, bar).

We can try generators in the erlang shell using the function eqc_gen:sample/1 which prints 11 values randomly generated by the generator given as input. it will be used in the next part of the documentation to make some examples of the generators shipped with eqc.

Basic generators

Quickcheck offers lots of basic generators out of the box:

eqc_gen:binary/0

Generates a binary of random size.

1> eqc_gen:sample(eqc_gen:binary()).
<<66,74,153,140>>
<<>>
<<>>
<<117,130>>
<<4,40>>
<<"æ¼n×S">>
<<>>
<<145>>
<<205,139,246,38,82,152>>
<<46,3,121,222,6,224>>
<<152>>

eqc_gen:binary/1

Generates a binary of a given size in bytes.

1> eqc_gen:sample(eqc_gen:binary(10)).
<<30,163,217,170,29,110,96,222,192,11>>
<<50,122,189,64,4,109,85,145,50,143>>
<<84,142,179,123,243,176,63,250,211,52>>
<<170,250,53,140,128,34,162,27,54,68>>
<<12,134,245,165,247,70,180,39,200,127>>
<<87,25,126,114,218,11,184,218,19,95>>
<<18,242,178,233,22,143,208,78,81,29>>
<<22,67,51,123,128,240,96,12,35,83>>
<<91,174,194,25,157,61,185,149,140,90>>
<<185,106,72,208,199,186,208,39,58,16>>
<<182,62,47,11,200,178,84,10,154,84>>

eqc_gen:bistring/0

Generates a list of bits in a bitstring. For Erlang release R12B and later. The bitstring shrinks both in size as well as in content. If you consider the bitstring as a representation of a number, then each shrinking step will result in a smaller-or-equal number.

1> eqc_gen:sample(eqc_gen:bitstring()).
<<1:1>>
<<77,6:3>>
<<3:3>>
<<123,13:5>>
<<186,102:7>>
<<1:2>>
<<11:5>>
<<230,217,0:1>>
<<232,51,1:1>>
<<152,25,1:2>>
<<"Ý°Þ">>

eqc_gen:bitstring/1

Generates a bitstring of a given size in bits. For Erlang release R12B and later. When shrinking, the size is unchanged, but content shrinks like eqc_gen:bitstring/0.

1> eqc_gen:sample(eqc_gen:bitstring(10)).
<<155,0:2>>
<<149,0:2>>
<<160,0:2>>
<<59,0:2>>
<<34,0:2>>
<<37,0:2>>
<<19,0:2>>
<<212,0:2>>
<<163,0:2>>
<<27,0:2>>
<<172,0:2>>

eqc_gen:bool/0

Generates a random boolean. Shrinks to false.

1> eqc_gen:sample(eqc_gen:bool()).
true
true
true
true
true
true
true
false
false
true
true

eqc_gen:char/0

Generates a random character. Shrinks to a, b or c.

1> eqc_gen:sample(eqc_gen:char()).
184
252
115
145
87
234
123
207
1
228
159

eqc_gen:choose/2

Generates a number in the range M to N. The result shrinks towards smaller absolute values.

1> eqc_gen:sample(eqc_gen:choose(10, 50)).
13
50
30
32
37
40
34
44
21
48
23

eqc_gen:elements/1

Generates an element of the list argument. Shrinking chooses an earlier element.

1> eqc_gen:sample(eqc_gen:elements([1, 2, 3, a, b, c, 1.2])).
3
3
1.2
1
1
1.2
b
2
b
a
c

eqc_gen:function0/1

Generates a function of no arguments with result generated by G.

1> eqc_gen:sample(eqc_gen:function0(eqc_gen:int())).
#Fun<eqc_gen.76.3938019>
#Fun<eqc_gen.76.3938019>
#Fun<eqc_gen.76.3938019>
#Fun<eqc_gen.76.3938019>
#Fun<eqc_gen.76.3938019>
#Fun<eqc_gen.76.3938019>
#Fun<eqc_gen.76.3938019>
#Fun<eqc_gen.76.3938019>
#Fun<eqc_gen.76.3938019>
#Fun<eqc_gen.76.3938019>
#Fun<eqc_gen.76.3938019>

eqc_gen:function1/1

Generates a function of one argument with result generated by G. The generated function is pure (will always return the same result for the same argument) and the result depends randomly on the argument.

1> eqc_gen:sample(eqc_gen:function1(eqc_gen:int())).
#Fun<eqc_gen.75.54864535>
#Fun<eqc_gen.75.54864535>
#Fun<eqc_gen.75.54864535>
#Fun<eqc_gen.75.54864535>
#Fun<eqc_gen.75.54864535>
#Fun<eqc_gen.75.54864535>
#Fun<eqc_gen.75.54864535>
#Fun<eqc_gen.75.54864535>
#Fun<eqc_gen.75.54864535>
#Fun<eqc_gen.75.54864535>
#Fun<eqc_gen.75.54864535>

eqc_gen:int/0

Generates a small integer (with absolute value bounded by the generation size).

1> eqc_gen:sample(eqc_gen:int()).
-3
10
-1
5
0
13
-4
-1
7
-14
-5

eqc_gen:largeint/0

Generates an integer from a large range.

1> eqc_gen:sample(eqc_gen:largeint()).
-8711186581
2766339275
8563728480
9752888411
-3617529945
-5321510976
3751872946
-7023579022
5797674866
-5773063098
641098985

eqc_gen:list/1

Generates a list of elements generated by its argument. Shrinking drops elements from the list. The length of the list varies up to one third of the generation size parameter.

1> eqc_gen:sample(eqc_gen:list(eqc_gen:int())).
[]
[5,-2,-8]
[]
[13,7,5,4]
[-6,-3]
[-10,5,13,-12]
[-6]
[]
[-13]
[-9,19,18,-5,6,8]
[-8]

eqc_gen:nat/0

Generates a small natural number (bounded by the generation size).

1> eqc_gen:sample(eqc_gen:nat()).
0
2
9
7
5
6
1
0
13
11
18

eqc_gen:orderedlist/1

Generates an ordered list of elements generated by G.

1> eqc_gen:sample(eqc_gen:orderedlist(eqc_gen:nat())).
[]
[7]
[1,10,10]
[7,11]
[0,2,6]
[5,8]
[8,8,14,15]
"\b"
[0,6,15,17]
[5,9,13,13,15,19,19]
[]

eqc_gen:real/0

Generates a real number.

1> eqc_gen:sample(eqc_gen:real()).
-6.0
1.75
-1.125
1.0
1.0
1.8571428571428572
3.0
-2.125
0.7222222222222222
0.3333333333333333
-1.5833333333333333

eqc_gen:return/1

Constructs a generator that always generates the value X. Most values can also be used as generators for themselves, making return unnecessary, but return(X) may be more efficient than using X as a generator, since when return(X) is used then QuickCheck does not traverse X searching for values to be intepreted specially.

1> eqc_gen:sample(eqc_gen:return(foo)).
foo
foo
foo
foo
foo
foo
foo
foo
foo
foo
foo

eqc_gen:vector/2

Generates a list of the given length, with elements generated by G.

1> eqc_gen:sample(eqc_gen:vector(4, eqc_gen:real())).
[-1.2,1.1428571428571428,-4.0,0.0]
[-0.6666666666666666,-0.1111111111111111,-1.0,-0.7142857142857143]
[2.0,-0.2,0.2727272727272727,0.75]
[0.0,-0.1,-3.25,0.8888888888888888]
[-6.0,-1.6666666666666667,2.2,-2.0]
[-1.125,-1.1818181818181819,0.13333333333333333,-0.8]
[1.4,-15.0,4.5,-0.7777777777777778]
[0.8235294117647058,-11.0,6.0,1.2]
[-0.375,3.0,-2.0,0.5833333333333334]
[0.3333333333333333,0.125,-1.7,0.17647058823529413]
[-1.0526315789473684,-1.6,-1.2222222222222223,-2.2857142857142856]

TODO for complete the eqc_gen API:

• eqc_gen:non_empty/1
• eqc_gen:oneof/1
• eqc_gen:resize/2
• eqc_gen:shuffle/1**

Custom generators

** warning **: this paragraph is incomplete and only shows an easy way to create a cusotm generator. It will be updated in the next weeks.

In some situations we need to define a custom generator, which is the composition of other generators. For example, suppose that we are testing an algorithm to solve the Knapsack problem.

From Wikipedia: Given a set of items, each with a mass and a value, determine the number of each item to include in a collection so that the total weight is less than or equal to a given limit and the total value is as large as possible.

So that it seems we need a generators for the items in order to do some property based testing on our algorithm.

An item is something with a mass and a value, so that in erlang we can model this with a map. Here is an example of item:

#{ mass => 5, value => 10 }.

How can we generate radom items?

Mass is a positive integer while the value is a natural number. This is arguable but it is out of the scope of this paragraph.

To create a custom generator we use the ?LET(Pat, G1, G2) macro which from the official documentation Generates a value from G1, binds it to Pat, then generates a value from G2 (which may refer to the variables bound in Pat). Bear in mind that weight can only be greater than or equal to 0. So that, a good idea could be to use the eqc_gen:nat/0 basic generator but it generates also the number 0. To avoid wrong generations we can use the ?SUCHTHAT(X, G, P) macro which from the offical documentation Generates values X from G such that the condition P is true.

Here is the code:

item() ->
  ?LET(
     {Weight, Value},
     {?SUCHTHAT(W, nat(), W > 0), nat()},
     #{ w => Weight, v => Value}).

We can read this as: "Generate a map with two keys: w and v. The value of the key w is a randomly generated natural number greater than 0 and the value of the key v is a randomly generated natural number".

And here are some samples of our custom generator (look at how we can test generators defined in our test modules, in this case knapsack_eqc):

1> eqc_gen:sample(knapsack_eqc:item()).
#{v => 2,w => 1}
#{v => 1,w => 7}
#{v => 3,w => 12}
#{v => 8,w => 8}
#{v => 10,w => 1}
#{v => 9,w => 4}
#{v => 15,w => 12}
#{v => 6,w => 17}
#{v => 10,w => 8}
#{v => 9,w => 8}
#{v => 15,w => 5}

Warning: it is too easy to get wrong in writing new generators, especially if you are new to QuickCheck. You can easily validate that at least you have written a generator and not something else. To do this check we can use the next eqc_gen function:

eqc_gen:is_generator/1

Returns true if the argument is a QuickCheck generator.

%% file knapsack_eqc
a_generator() ->
  ?LET(
     {Weight, Value},
     {?SUCHTHAT(W, nat(), W > 0), nat()},
     #{ w => Weight, v => Value}).

not_a_generator() ->
  #{ w => nat(), v => nat() }.

%% shell
1> eqc_gen:is_generator(knapsack_eqc:a_generator()).
true
2> eqc_gen:is_generator(knapsack_eqc:not_a_generator()).
false

Think shrinking

Since we have explained generators it is time to introduce the term shrinking. What does it means?

Basically shrinking means: getting physically smaller and in the Quickcheck world it means that when an input which violates a property has been found then Quickcheck will try to find smaller inputs that also violate the property in order to help the developer with a better error message.

The meaning of small in this case depends on the datatype of the generator and to understand it there is nothing better than a real example, we can try how a generator shrinks with the function eqc_gen:sampleshrink/1 which is really useful when we are dealing with custom generators.

eqc_gen:sampleshrink/1

Prints a value generated by G, followed by one way of shrinking it. Each following line displays a list of values that the first value on the previous line can be shrunk to in one step. Thus the output traces the leftmost path through the shrinking tree.

1> eqc_gen:sampleshrink(knapsack_eqc:thing()).
#{v=>7,w=>7}
--> [#{v=>7,w=>2},#{v=>7,w=>6},#{v=>0,w=>7},#{v=>2,w=>7},#{v=>6,w=>7}]
--> [#{v=>7,w=>1},#{v=>0,w=>2},#{v=>2,w=>2},#{v=>6,w=>2}]
--> [#{v=>0,w=>1},#{v=>2,w=>1},#{v=>6,w=>1}]