Introduction

Recently I was invited by Talk Julia to take part in the podcast. Today it has been released and you can watch it on YouTube. In the discussion I share my thoughts on DataFrames.jl design principles and discuss several examples from my upcoming Julia for Data Analysis book.

One of the things I discuss in the podcast is that in DataFrames.jl development process we are putting a lot of emphasis on tests so I thought that it is worth to expand on this topic a bit more in this post.

The codes I use were tested under Julia 1.7.2 and InlineStrings.jl 1.2.2.

Testing and reproducibility of tests

When one develops some software ensuring its proper test coverage is one of the key practices that help maintaining good quality of code.

My experience with DataFrames.jl is that taking care of proper test coverage serves three important goals:

  1. Making sure the functionality we provide follows the contract specified in its documentation. This is particularly important when testing corner case situations, since they happen rarely in practice (so there is lower probability that users spot problems and report them) and also allow us to make sure our design is logically consistent.
  2. Making sure that as we add new functionalities to the package we do not accidentally break something.
  3. Making sure that upgrading dependencies of DataFrames.jl to newer versions does not break something (and the biggest such dependency is Base Julia).

A crucial part of writing tests is ensuring their reproducibility. What I mean by this is that when you run your test suite twice on you should get the same results. This intends to avoid situations in which you run your tests and get a bug report. Then you run the tests again and the bug is not present. Such situation is unwanted, as it is later hard to locate the root cause of the reported bug.

Randomized tests

When one implements complex algorithms it is hard to cover all possible hard testing scenarios by writing them down by hand. In such situations, one of the possible testing methods is to use randomized tests.

An example, old and already resolved, problem that potentially could have been caught by randomized tests is an issue related to pasting Unicode characters in Julia REPL. Let me here present a minimal working example of a code having a similar problem.

Assume I want to write a code that strips last character from a string. Here is an attempt to implement it:

julia> mychop(s::AbstractString) = isempty(s) ? s : s[1:end-1]
mychop (generic function with 1 method)

Let us write a simple test set for this function:

julia> using Test

julia> @testset "basic test" begin
           @test mychop("") == ""
           @test mychop("a") == ""
           @test mychop("abc") == "ab"
       end
Test Summary: | Pass  Total
basic test    |    3      3
Test.DefaultTestSet("basic test", Any[], 3, false, false)

All looks good so far. However, let us run some more advanced testing of mychop using randomized tests:

julia> using Random

julia> @testset "advanced tests" begin
           Random.seed!(1234)
           for _ in 1:40
               len = rand(1:10)
               input = rand(UInt8, len) |> String
               output = join(collect(input)[1:end-1])
               @test output == mychop(input)
           end
       end
advanced tests: Error During Test at REPL[83]:7
  Test threw exception
  Expression: output == mychop(input)
  StringIndexError: invalid index [9], valid nearby indices [8]=>'˄', [10]=>'�'
Test Summary:  | Pass  Error  Total
advanced tests |   39      1     40
ERROR: Some tests did not pass: 39 passed, 0 failed, 1 errored, 0 broken.

What I do in this test is generateing input string using randoom bits and then use a slow method to get a desired output string by going through individual characters of the original string. I ensure reproducibility of this test on a given version of Julia by setting the seed of random number generator using Random.seed!(1234).

From the bug report we can see that something is not right with indexing. The problem occurs if the second character from the end of the string is not ASCII. Let us check it:

julia> mychop("∀a")
ERROR: StringIndexError: invalid index [3], valid nearby indices [1]=>'∀', [4]=>'a'

The point of randomized test is that guessing the test scenario second character from the end of the string is not ASCII is not easy.

Let us fix mychop to resolve this problem:

julia> mychop(s::AbstractString) = isempty(s) ? s : s[1:prevind(s, end)]
mychop (generic function with 1 method)

julia> @testset "basic test" begin
           @test mychop("") == ""
           @test mychop("a") == ""
           @test mychop("abc") == "ab"
       end
Test Summary: | Pass  Total
basic test    |    3      3
Test.DefaultTestSet("basic test", Any[], 3, false, false)

julia> @testset "advanced tests" begin
           Random.seed!(1234)
           for _ in 1:32768 # pick some round larger number to be sure all works well
               len = rand(1:10)
               input = rand(UInt8, len) |> String
               output = join(collect(input)[1:end-1])
               @test output == mychop(input)
           end
       end
Test Summary:  |  Pass  Total
advanced tests | 32768  32768
Test.DefaultTestSet("advanced tests", Any[], 32768, false, false)

Are we done now? Not yet. We should check our function with some other AbstractString than String. Let me use InlineStrings.jl:

julia> using InlineStrings

julia> @testset "InlineStrings.jl test" begin
           @test "ab" == @inferred mychop(InlineString("abc"))
       end
InlineStrings.jl test: Error During Test at REPL[110]:2
  Test threw exception
  Expression: "ab" == #= REPL[110]:2 =# @inferred(mychop(InlineString("abc")))
  return type SubString{String3} does not match inferred return type Union{SubString{String3}, String3}
Test Summary:         | Error  Total
InlineStrings.jl test |     1      1
ERROR: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.

As you can see there is still some more work to be done, as our mychop function is not type stable, which is checked by the @inferred macro. The problem is that indexing into String3 that happens if the passed string is not empty produces a SubString. Let us fix it by making sure that in every branch of code we apply the same operation to our source string (in the code, like in the codes above, we use the fact that AbstractString is guaranteed to use 1-based indexing).

julia> mychop(s::AbstractString) = s[1:prevind(s, max(1, end))]
mychop (generic function with 1 method)

julia> @testset "advanced tests" begin
           Random.seed!(1234)
           for _ in 1:32768 # pick some round larger number to be sure all works well
               len = rand(0:10) # make sure to cover 0-length strings
               input = rand(UInt8, len) |> String
               output = join(collect(input)[1:end-1])
               @test output == @inferred mychop(input)
               @test output == @inferred mychop(InlineString(input))
           end
       end
Test Summary:  |  Pass  Total
advanced tests | 65536  65536
Test.DefaultTestSet("advanced tests", Any[], 65536, false, false)

Now all works as expected and we are done.

Conclusions

I hope you found my thoughts on writing tests useful. As usual, I have some additional comments regarding the presented codes:

  • I used Random.seed! explicitly in my tests. However, the @testset macro, before the execution of its body, makes a call to Random.seed!(seed) where seed is the current seed of the global RNG. Therefore, if you design a larger test suite you do not have to set the seed in every @testset. It is enough to set it once per all tests you run.
  • As I have already commented, the presented codes are reproducible on a given version of Julia. If you want them to be reproducible across different versions of Julia I recommend you to use for example the StableRNGs.jl package as a source of randomness in your tests.