Standard library · System and tools

testing

Writing tests for your code: equality assertions, a pass and fail count, and a summary at the end.

import testing

A test is a small program that checks your code does what it should. The testing package has assertions that print a clear message when they fail but do not stop the program, so every check runs and Summary says at the end how many passed and how many failed. Put tests in a file whose name ends in _test.salam, write each one as a function whose name starts with test_, and run them all with salam test, locally or in CI (see below). A test file can also be an ordinary program with its own main, started with salam run.

Summary returns the number of failures, and returning it from main makes the program end with a non-zero code when something failed, which is what turns a CI job red. For checks inside the program itself, conditions that must never be false in production code, use Assert from debug instead, which stops the program. To time code, see debug and time.

The things people get wrong:

  • A failure line says what differed, not where. In a loop over cases, use AssertMsg and put the case name in the message (see below).
  • Strings are printed without quotes, so a stray space is invisible in got hello want hello. Put brackets around values in your own messages.
  • An exit code only has 8 bits, so 256 failures read as exit code 0. A suite that can have that many should return failures > 0 ? 1 : 0.
  • The counters are plain globals with no lock: call the assertions from the main thread only.

Running tests with salam test

salam test finds every *_test.salam file under the current directory (or the files and directories you name), and in each one calls every function whose name starts with test_ (آزمون in a Persian file) and that takes no arguments. You do not write main: the runner writes its own, in memory, and reports each test by name:

salam test                      # every test file under this directory
salam test calc_test.salam      # one file
salam test --run=parse          # only tests whose name contains "parse"
salam test '--run=^test_add$'   # exactly test_add (^ and $ anchor the name)
salam test --interp             # on the interpreter, no C compiler needed

Each file runs in its own process, so a crash fails that file and the others still run. The command ends with exit code 1 when any test failed, when a test file does not compile, and when --run matches no test at all. --format=teamcity prints TeamCity service messages instead of plain lines, which is what the JetBrains plugin reads to show the test tree; a file that has its own main and no test_ functions is run as one test that passes when it exits with 0.

Table driven tests

When one function has to give the right answer for many inputs, put the inputs and the expected answers in an array of small structs and loop over it. Adding a case is then one line, and every case runs even when an earlier one fails.

testing-guide-table.salam
import testing
import str

struct Case:
    pub n: int = 0
    pub want: str = ""
end

func fizzbuzz(n: int): str:
    if n % 15 == 0:
        ret "FizzBuzz"
    else n % 3 == 0:
        ret "Fizz"
    else n % 5 == 0:
        ret "Buzz"
    end
    ret str.FromInt(n)
end

func main:
    cases := [
        Case { n = 1, want = "1" },
        Case { n = 3, want = "Fizz" },
        Case { n = 5, want = "Buzz" },
        Case { n = 9, want = "Fizz" },
        Case { n = 10, want = "Buzz" },
        Case { n = 15, want = "FizzBuzz" },
        Case { n = 98, want = "98" }
    ]
    each c in cases:
        testing.AssertEqStr(fizzbuzz(c.n), c.want)
    end
    ret testing.Summary()
end
Output-- tests: 7 passed, 0 failed

The program returns what Summary returns, so it ends with exit code 0 when everything passed. Run it like any other program:

salam run fizzbuzz_test.salam
echo $?

Failure messages that point at the case

AssertEqStr inside a loop prints strings differ: got ... want ..., which leaves you to work out which row of the table it was. Give each case a name and build the message yourself with AssertMsg. Brackets around the values show spaces at the edges, which is exactly the bug in this example.

testing-guide-labels.salam
import testing
import str

struct Case:
    pub name: str = ""
    pub input: str = ""
    pub want: str = ""
end

func slug(title: str): str:
    ret str.Replace(str.Lower(title), " ", "-")
end

func check(c: Case):
    got := slug(c.input)
    testing.AssertMsg(got == c.want, c.name + ": got [" + got + "] want [" + c.want + "]")
end

func main:
    cases := [
        Case { name = "plain words", input = "Hello World", want = "hello-world" },
        Case { name = "trailing space", input = "Release notes ", want = "release-notes" },
        Case { name = "one word", input = "Salam", want = "salam" }
    ]
    each c in cases:
        check(c)
    end
    failures := testing.Summary()
    println "exit code:", failures > 0 ? 1 : 0
end
OutputFAIL: trailing space: got [release-notes-] want [release-notes] -- tests: 2 passed, 1 failed exit code: 1

Group the tests of each function

A long test file is easier to read when each function under test gets its own test function. Failed counts every failure so far; remember it at the start of a group and compare at the end to report the group as a whole.

testing-guide-groups.salam
import testing

func clamp(v: int, low: int, high: int): int:
    if v < low:
        ret low
    end
    if v > high:
        ret high
    end
    ret v
end

func average(values: Vector<int>): f64:
    if values.Len() == 0:
        ret 0.0
    end
    mut total := 0
    each v in values:
        total += v
    end
    ret total as f64 / values.Len() as f64
end

func report(name: str, failed before: int):
    if testing.Failed() == failed before:
        println "ok  ", name
    else:
        println "FAIL", name
    end
end

func test clamp():
    before := testing.Failed()
    testing.AssertEqInt(clamp(5, 0, 10), 5)
    testing.AssertEqInt(clamp(-3, 0, 10), 0)
    testing.AssertEqInt(clamp(42, 0, 10), 10)
    report("clamp", before)
end

func test average():
    before := testing.Failed()
    mut v := Vector {} as Vector<int>
    defer v.Free()
    testing.AssertEqFloat(average(v), 0.0, 0.000001)
    v.Push(1)
    v.Push(2)
    v.Push(4)
    testing.AssertEqFloat(average(v), 2.333333, 0.00001)
    report("average", before)
end

func main:
    test clamp()
    test average()
    ret testing.Summary()
end
Outputok clamp ok average -- tests: 5 passed, 0 failed

Floats are compared with AssertEqFloat and a tolerance, never with ==: 7 / 3 has no exact binary form, so the last digits differ from any decimal you write down.

Test the error paths too

The code that runs when something goes wrong is the code that is run least, so test it on purpose. For a function that reports problems through an err &: out parameter, call it with bad input and check both the result and the message. AssertContains checks for a key word, so the test keeps passing when the wording around it is improved.

testing-guide-errors.salam
import testing

func parse port(text: str, err &: str): int:
    err = ""
    if text.Len() == 0:
        err = "the port is empty"
        ret 0
    end
    n := text.ToInt()
    if n < 1 or n > 65535:
        err = "the port must be between 1 and 65535"
        ret 0
    end
    ret n
end

func main:
    mut err := ""
    testing.AssertEqInt(parse port("8080", err), 8080)
    testing.AssertEqStr(err, "")

    testing.AssertEqInt(parse port("", err), 0)
    testing.AssertContains(err, "empty")

    testing.AssertEqInt(parse port("70000", err), 0)
    testing.AssertNeStr(err, "")
    testing.AssertContains(err, "65535")
    ret testing.Summary()
end
Output-- tests: 7 passed, 0 failed

What a failure looks like

A failed assertion prints a FAIL: line with what it got and what it wanted, and the program carries on with the next check. The failure lines go to standard error and the summary to standard output, so a CI log shows both but a pipe can separate them.

testing-guide-failure.salam
import testing

func initials(name: str): str:
    parts := name.Split(" ")
    mut out := ""
    each p in parts:
        out += p.Substr(0, 1)
    end
    ret out
end

func main:
    testing.AssertEqStr(initials("Ada Lovelace"), "AL")
    testing.AssertEqStr(initials("grace hopper"), "GH")
    testing.AssertEqInt(testing.Failed(), 1)
    failures := testing.Summary()
    println "exit code a CI job would see:", failures
end
OutputFAIL: strings differ: got gh want GH -- tests: 2 passed, 1 failed exit code a CI job would see: 1

This example prints the failure count instead of returning it, so the page can show its output. A real test file ends with ret testing.Summary(), and the non-zero exit code is what makes the CI job fail.

Which assertion to use

You are checkingUseWhy
Two whole numbersAssertEqInt, AssertEqInt64The message shows both values
Two stringsAssertEqStrThe message shows both strings
FloatsAssertEqFloat with a small toleranceRounding makes == unreliable
Part of a longer textAssertContainsSurvives changes to the rest of the text
That something changedAssertNeInt, AssertNeStrFor ids, tokens and hashes
A pointer was or was not setAssertNil, AssertNotNilFor C libraries and allocations
Any other conditionAssertMsgSay in words what went wrong

Prefer the specific assertions to Assert. "ints differ: got 4 want 5" tells you far more than "assertion failed". Reset sets both counters back to zero, for a program that runs several separate suites in a row.

Functions

Assert

func Assert(cond: bool)
Records a failure, printed as FAIL: assertion failed, when the condition is false. Prefer an assertion that says more.
testing-Assert.salam
import mem
import testing

func add(a: int, b: int): int:
    ret a + b
end

func main:
    testing.Reset()
    testing.Assert(add(2, 2) == 4)
    testing.AssertMsg(add(1, 1) == 2, "one and one")
    testing.AssertTrue(true)
    testing.AssertFalse(add(0, 0) == 1)
    testing.AssertEqInt(add(3, 4), 7)
    testing.AssertEqInt64(10000000000 as i64, 10000000000 as i64)
    testing.AssertEqFloat(0.1 + 0.2, 0.3, 0.0001)
    testing.AssertContains("the Salam language", "Salam")
    testing.AssertEqStr("hello", "hello")
    testing.AssertEqBool(1 < 2, true)
    testing.AssertNeInt(1, 2)
    testing.AssertNeStr("a", "b")
    testing.AssertNil(null)
    block := mem.Allocate(8 as u64)
    testing.AssertNotNil(block)
    mem.Free(block)
    testing.AssertEqInt(add(2, 2), 5)
    println testing.Passed(), testing.Failed()
    println testing.Summary()
end
OutputFAIL: ints differ: got 4 want 5 14 1 -- tests: 14 passed, 1 failed 1

AssertMsg

func AssertMsg(cond: bool, msg: str)
Like Assert, with your own message after FAIL: .

The example for this item is shown with the ones above: testing-Assert.salam

AssertTrue

func AssertTrue(cond: bool)
The condition has to be true.

The example for this item is shown with the ones above: testing-Assert.salam

AssertFalse

func AssertFalse(cond: bool)
The condition has to be false.

The example for this item is shown with the ones above: testing-Assert.salam

AssertEqInt

func AssertEqInt(got: int, want: int)
Two whole numbers have to be equal; on failure both values are printed.

The example for this item is shown with the ones above: testing-Assert.salam

AssertEqInt64

func AssertEqInt64(got: i64, want: i64)
Two 64-bit whole numbers have to be equal.

The example for this item is shown with the ones above: testing-Assert.salam

AssertEqFloat

func AssertEqFloat(got: f64, want: f64, eps: f64)
Two decimal numbers may differ by at most eps. Never compare decimals with ==.

The example for this item is shown with the ones above: testing-Assert.salam

AssertContains

func AssertContains(hay: str, needle: str)
The first string has to contain the second; on failure both are printed.

The example for this item is shown with the ones above: testing-Assert.salam

AssertEqStr

func AssertEqStr(got: str, want: str)
Two strings have to be equal; on failure both are printed, without quotes.

The example for this item is shown with the ones above: testing-Assert.salam

AssertEqBool

func AssertEqBool(got: bool, want: bool)
Two booleans have to be equal. The failure message does not show the values.

The example for this item is shown with the ones above: testing-Assert.salam

AssertNeInt

func AssertNeInt(a: int, b: int)
Two whole numbers have to differ. The failure message does not show the value.

The example for this item is shown with the ones above: testing-Assert.salam

AssertNeStr

func AssertNeStr(a: str, b: str)
Two strings have to differ. The failure message does not show the value.

The example for this item is shown with the ones above: testing-Assert.salam

AssertNil

func AssertNil(p: void*)
The pointer has to be null.

The example for this item is shown with the ones above: testing-Assert.salam

AssertNotNil

func AssertNotNil(p: void*)
The pointer must not be null.

The example for this item is shown with the ones above: testing-Assert.salam

Reset

func Reset()
Sets the counters back to zero.

The example for this item is shown with the ones above: testing-Assert.salam

Passed

func Passed(): int
The number of assertions that passed.

The example for this item is shown with the ones above: testing-Assert.salam

Failed

func Failed(): int
The number of assertions that failed.

The example for this item is shown with the ones above: testing-Assert.salam

Summary

func Summary(): int
Prints -- and tests: N passed, M failed to standard output and returns the number of failures.

The example for this item is shown with the ones above: testing-Assert.salam

SetFormat

func SetFormat(format: str)
Chooses how Begin, End and Finish report: "teamcity" prints TeamCity service messages, which JetBrains IDEs and CI servers read as a test tree; anything else prints plain lines. salam test calls it for you.
testing-Runner.salam
import testing

func add(a: int, b: int): int:
    ret a + b
end

func main:
    testing.SetFormat("human")
    testing.SetFilter("add")
    testing.SuiteStart("testing-Runner.salam")
    if testing.Selected("test_add"):
        testing.Begin("test_add", 12)
        testing.AssertEqInt(add(2, 2), 4)
        testing.End()
    end
    if testing.Selected("test_add_zero"):
        testing.Begin("test_add_zero", 17)
        testing.AssertEqInt(add(5, 0), 5)
        testing.End()
    end
    testing.Finish()
end
Output=== RUN test_add --- PASS: test_add === RUN test_add_zero --- PASS: test_add_zero ok testing-Runner.salam: 2 tests passed

SetFilter

func SetFilter(part: str)
Runs only the tests whose name contains the given text; salam test --run=... sets it.

The example for this item is shown with the ones above: testing-Runner.salam

Selected

func Selected(name: str): bool
Whether a test with this name runs under the current filter.

The example for this item is shown with the ones above: testing-Runner.salam

SuiteStart

func SuiteStart(file: str)
Starts the report for one test file. salam test calls it for you.

The example for this item is shown with the ones above: testing-Runner.salam

Begin

func Begin(name: str, line: int)
Starts one test, with its name and the line it is declared on. Failures recorded until End belong to it.

The example for this item is shown with the ones above: testing-Runner.salam

End

func End()
Ends the current test and reports it as passed or failed.

The example for this item is shown with the ones above: testing-Runner.salam

Finish

func Finish()
Reports the whole file and ends the program with exit code 1 when a test failed.

The example for this item is shown with the ones above: testing-Runner.salam