JCheck.jl Documentation

What is JCheck.jl?

JCheck is a test framework for the Julia programming language. It aims to replicate some of the functionalities of Quickcheck. The user specifies a set of properties as predicates, and JCheck then attempts to find cases that violate these predicates. Since it is generally impossible to evaluate a predicate for every possible input, JCheck, like QuickCheck, uses a Monte Carlo approach: it generates a set of random inputs and passes them as arguments to the predicates. Serialization to a JLSO file is enabled by default to facilitate analysis of problematic cases.

Features

• Reuse inputs to reduce the time spent on case generation.
• Serialization of problematic cases for easier analysis.
• Integration with Julia's testing framework.
• Allow specification of "special cases," that is, non-random inputs that are always checked.
• Shrinkage of failing test cases.

Usage

Container

Predicates must be contained in a Quickcheck object to be used in a test. They are easy to create. The most basic way is to call the constructor with a brief and simple description:

julia> using Test: @testset, @test
julia> using JCheck
julia> qc = Quickcheck("A Test")A Test: 0 predicate and 0 free variable.

For more advanced uses, see the documentation of the Quickcheck constructor.

Adding predicates

Once a Quickcheck object has been created, the next step is to populate it with predicates. This can be done using the @add_predicate macro.

julia> @add_predicate qc "Sum commute" ((x::Float64, n::Int) -> x + n == n + x)A Test: 1 predicate and 2 free variables:
n::Int64
x::Float64

A predicate is a function that returns either true or false. In the context of JCheck, the form of the predicate is strict; please read the documentation of @add_predicate.

(Quick)checking

The macro @quickcheck initiates the process of searching for falsifying instances within a Quickcheck object.

julia> @quickcheck qcTest Summary:    | Pass  Total  Time
Test Sum commute |    1      1  0.4s

As part of a @testset

The @quickcheck macro can be nested within a @testset. This facilitates integration into a package's test suite.

@testset "Sample test set" begin
    @test isempty([])

    @quickcheck qc
end

Test Summary:   | Pass  Total
Sample test set |    2      2

Let's add a failing predicate.

@add_predicate qc "I fail" (x::Float64 -> false)

@testset "Sample failing test set" begin
    @test isempty([])

    @quickcheck qc
end

┌ Warning: Predicate "I fail" does not hold for valuation (x = 0.0,)
└ @ JCheck ~/Projets/JCheck/src/Quickcheck.jl:267
┌ Warning: Predicate "I fail" does not hold for valuation (x = 1.0,)
└ @ JCheck ~/Projets/JCheck/src/Quickcheck.jl:267

[...]

Some predicates do not hold for some valuations; they have been saved
to JCheck_yyyy-mm-dd_HH-MM-SS.jchk. Use function load and macro @getcases
to explore problematic cases.

Test Summary:           | Pass  Fail  Total
Sample failing test set |    2     1      3
  Test Sum commute      |    1            1
  Test I fail           |          1      1
ERROR: Some tests did not pass: 2 passed, 1 failed, 0 errored, 0 broken.

Analysing failing cases

By default, failing test cases are serialized to a JLSO file for subsequent analysis.

julia> ft = JCheck.load("JCheck_test.jchk")2 failing predicates:
Is odd
- commutes

Failing cases for a predicate can be extracted using its description with @getcases. There's no need to provide the exact description of the predicate you want to extract; the entry whose description is closest (in terms of Levenshtein distance) will be matched.

julia> pred, valuations = @getcases ft i od@NamedTuple{predicate::Function, valuations::Vector{Tuple}}((Serialization.__deserialized_types__.var"##285"(), Tuple[(0,), (-9223372036854775808,), (-1603514452799603314,), (1420394807175553538,), (4507329505808279390,), (-426481527288535688,), (-5691388592443778052,), (-7859122130299025792,), (-5525456812138927418,), (-7209867710197627164,)  …  (2031324158527932024,), (7907216074681153692,), (4734352501972781814,), (7649976476383282706,), (-6664068458754296008,), (-5721291110713069694,), (8573438617342549320,), (-5611383820228536680,), (-4303975626508744234,), (-5727584619371173840,)]))
julia> map(x -> pred(x...), valuations) # each element of `valuations` is a tuple.53-element Vector{Bool}:
 0
 0
 0
 0
 0
 0
 0
 0
 0
 0
 ⋮
 0
 0
 0
 0
 0
 0
 0
 0
 0

Types with built-in generators

For a list of types for which a generator is included in the package, see the reference for generate.

Testing With Custom Types

JCheck can be easily extended to work with custom types from which it is possible to randomly generate instances. The only requirement is to overload generate. For example, an implementation for the type Int64 could look like this:

julia> using Random: AbstractRNG
julia> generate(rng::AbstractRNG, ::Type{Int64}, n::Int) = rand(rng, Int64, n)generate (generic function with 1 method)

Optionally, it is possible to specify so-called "special cases" for a type. These are always checked. Implementing them is as easy as overloading specialcases. For Int, this could look like this:

julia> specialcases(::Type{Int64}) = Int64[0, 1, typemin(Int64), typemax(Int64)]specialcases (generic function with 1 method)

For implementation details, refer to the documentation of these two functions.

Shrinkage

@quickcheck](@ref) will attempt to shrink any failing test case if possible. In order to enable shrinkage for a given type, the following two methods must be implemented:

• shrinkable
• shrink

The first one is a predicate that evaluates to true for an object if it can be shrunk. The second is a function that returns a Vector of shrunk objects. The implementation for type AbstractString is as follows:

shrinkable(x::AbstractString) = length(x) >= 2

function shrink(x::AbstractString)
    shrinkable(x) || return typeof(x)[x]

    n = length(x) ÷ 2
    [x[1:n], x[range(n + 1, end)]]
end

shrink (generic function with 1 method)

For more details and a list of default shrinkers, see the documentation for these methods.