testing
Writing tests for your code: equality assertions, a pass and fail count, and a summary at the end.
import testing
A test is a small program that checks your code does what it should. The testing package has assertions that print a clear message when they fail but do not stop the program, so every check runs and Summary says at the end how many passed and how many failed. Put tests in a file whose name ends in _test.salam, write each one as a function whose name starts with test_, and run them all with salam test, locally or in CI (see below). A test file can also be an ordinary program with its own main, started with salam run.
Summary returns the number of failures, and returning it from main makes the program end with a non-zero code when something failed, which is what turns a CI job red. For checks inside the program itself, conditions that must never be false in production code, use Assert from debug instead, which stops the program. To time code, see debug and time.
The things people get wrong:
- A failure line says what differed, not where. In a loop over cases, use
AssertMsgand put the case name in the message (see below). - Strings are printed without quotes, so a stray space is invisible in
got hello want hello. Put brackets around values in your own messages. - An exit code only has 8 bits, so 256 failures read as exit code 0. A suite that can have that many should return
failures > 0 ? 1 : 0. - The counters are plain globals with no lock: call the assertions from the main thread only.
Running tests with salam test
salam test finds every *_test.salam file under the current directory (or the files and directories you name), and in each one calls every function whose name starts with test_ (آزمون in a Persian file) and that takes no arguments. You do not write main: the runner writes its own, in memory, and reports each test by name:
salam test # every test file under this directory
salam test calc_test.salam # one file
salam test --run=parse # only tests whose name contains "parse"
salam test '--run=^test_add$' # exactly test_add (^ and $ anchor the name)
salam test --interp # on the interpreter, no C compiler needed
Each file runs in its own process, so a crash fails that file and the others still run. The command ends with exit code 1 when any test failed, when a test file does not compile, and when --run matches no test at all. --format=teamcity prints TeamCity service messages instead of plain lines, which is what the JetBrains plugin reads to show the test tree; a file that has its own main and no test_ functions is run as one test that passes when it exits with 0.
Table driven tests
When one function has to give the right answer for many inputs, put the inputs and the expected answers in an array of small structs and loop over it. Adding a case is then one line, and every case runs even when an earlier one fails.
import testing
import str
struct Case:
pub n: int = 0
pub want: str = ""
end
func fizzbuzz(n: int): str:
if n % 15 == 0:
ret "FizzBuzz"
else n % 3 == 0:
ret "Fizz"
else n % 5 == 0:
ret "Buzz"
end
ret str.FromInt(n)
end
func main:
cases := [
Case { n = 1, want = "1" },
Case { n = 3, want = "Fizz" },
Case { n = 5, want = "Buzz" },
Case { n = 9, want = "Fizz" },
Case { n = 10, want = "Buzz" },
Case { n = 15, want = "FizzBuzz" },
Case { n = 98, want = "98" }
]
each c in cases:
testing.AssertEqStr(fizzbuzz(c.n), c.want)
end
ret testing.Summary()
endThe program returns what Summary returns, so it ends with exit code 0 when everything passed. Run it like any other program:
salam run fizzbuzz_test.salam
echo $?
Failure messages that point at the case
AssertEqStr inside a loop prints strings differ: got ... want ..., which leaves you to work out which row of the table it was. Give each case a name and build the message yourself with AssertMsg. Brackets around the values show spaces at the edges, which is exactly the bug in this example.
import testing
import str
struct Case:
pub name: str = ""
pub input: str = ""
pub want: str = ""
end
func slug(title: str): str:
ret str.Replace(str.Lower(title), " ", "-")
end
func check(c: Case):
got := slug(c.input)
testing.AssertMsg(got == c.want, c.name + ": got [" + got + "] want [" + c.want + "]")
end
func main:
cases := [
Case { name = "plain words", input = "Hello World", want = "hello-world" },
Case { name = "trailing space", input = "Release notes ", want = "release-notes" },
Case { name = "one word", input = "Salam", want = "salam" }
]
each c in cases:
check(c)
end
failures := testing.Summary()
println "exit code:", failures > 0 ? 1 : 0
endGroup the tests of each function
A long test file is easier to read when each function under test gets its own test function. Failed counts every failure so far; remember it at the start of a group and compare at the end to report the group as a whole.
import testing
func clamp(v: int, low: int, high: int): int:
if v < low:
ret low
end
if v > high:
ret high
end
ret v
end
func average(values: Vector<int>): f64:
if values.Len() == 0:
ret 0.0
end
mut total := 0
each v in values:
total += v
end
ret total as f64 / values.Len() as f64
end
func report(name: str, failed before: int):
if testing.Failed() == failed before:
println "ok ", name
else:
println "FAIL", name
end
end
func test clamp():
before := testing.Failed()
testing.AssertEqInt(clamp(5, 0, 10), 5)
testing.AssertEqInt(clamp(-3, 0, 10), 0)
testing.AssertEqInt(clamp(42, 0, 10), 10)
report("clamp", before)
end
func test average():
before := testing.Failed()
mut v := Vector {} as Vector<int>
defer v.Free()
testing.AssertEqFloat(average(v), 0.0, 0.000001)
v.Push(1)
v.Push(2)
v.Push(4)
testing.AssertEqFloat(average(v), 2.333333, 0.00001)
report("average", before)
end
func main:
test clamp()
test average()
ret testing.Summary()
endFloats are compared with AssertEqFloat and a tolerance, never with ==: 7 / 3 has no exact binary form, so the last digits differ from any decimal you write down.
Test the error paths too
The code that runs when something goes wrong is the code that is run least, so test it on purpose. For a function that reports problems through an err &: out parameter, call it with bad input and check both the result and the message. AssertContains checks for a key word, so the test keeps passing when the wording around it is improved.
import testing
func parse port(text: str, err &: str): int:
err = ""
if text.Len() == 0:
err = "the port is empty"
ret 0
end
n := text.ToInt()
if n < 1 or n > 65535:
err = "the port must be between 1 and 65535"
ret 0
end
ret n
end
func main:
mut err := ""
testing.AssertEqInt(parse port("8080", err), 8080)
testing.AssertEqStr(err, "")
testing.AssertEqInt(parse port("", err), 0)
testing.AssertContains(err, "empty")
testing.AssertEqInt(parse port("70000", err), 0)
testing.AssertNeStr(err, "")
testing.AssertContains(err, "65535")
ret testing.Summary()
endWhat a failure looks like
A failed assertion prints a FAIL: line with what it got and what it wanted, and the program carries on with the next check. The failure lines go to standard error and the summary to standard output, so a CI log shows both but a pipe can separate them.
import testing
func initials(name: str): str:
parts := name.Split(" ")
mut out := ""
each p in parts:
out += p.Substr(0, 1)
end
ret out
end
func main:
testing.AssertEqStr(initials("Ada Lovelace"), "AL")
testing.AssertEqStr(initials("grace hopper"), "GH")
testing.AssertEqInt(testing.Failed(), 1)
failures := testing.Summary()
println "exit code a CI job would see:", failures
endThis example prints the failure count instead of returning it, so the page can show its output. A real test file ends with ret testing.Summary(), and the non-zero exit code is what makes the CI job fail.
Which assertion to use
| You are checking | Use | Why |
|---|---|---|
| Two whole numbers | AssertEqInt, AssertEqInt64 | The message shows both values |
| Two strings | AssertEqStr | The message shows both strings |
| Floats | AssertEqFloat with a small tolerance | Rounding makes == unreliable |
| Part of a longer text | AssertContains | Survives changes to the rest of the text |
| That something changed | AssertNeInt, AssertNeStr | For ids, tokens and hashes |
| A pointer was or was not set | AssertNil, AssertNotNil | For C libraries and allocations |
| Any other condition | AssertMsg | Say in words what went wrong |
Prefer the specific assertions to Assert. "ints differ: got 4 want 5" tells you far more than "assertion failed". Reset sets both counters back to zero, for a program that runs several separate suites in a row.
Functions
Assert
func Assert(cond: bool)FAIL: assertion failed, when the condition is false. Prefer an assertion that says more.import mem
import testing
func add(a: int, b: int): int:
ret a + b
end
func main:
testing.Reset()
testing.Assert(add(2, 2) == 4)
testing.AssertMsg(add(1, 1) == 2, "one and one")
testing.AssertTrue(true)
testing.AssertFalse(add(0, 0) == 1)
testing.AssertEqInt(add(3, 4), 7)
testing.AssertEqInt64(10000000000 as i64, 10000000000 as i64)
testing.AssertEqFloat(0.1 + 0.2, 0.3, 0.0001)
testing.AssertContains("the Salam language", "Salam")
testing.AssertEqStr("hello", "hello")
testing.AssertEqBool(1 < 2, true)
testing.AssertNeInt(1, 2)
testing.AssertNeStr("a", "b")
testing.AssertNil(null)
block := mem.Allocate(8 as u64)
testing.AssertNotNil(block)
mem.Free(block)
testing.AssertEqInt(add(2, 2), 5)
println testing.Passed(), testing.Failed()
println testing.Summary()
endAssertMsg
func AssertMsg(cond: bool, msg: str)Assert, with your own message after FAIL: .The example for this item is shown with the ones above: testing-Assert.salam
AssertTrue
func AssertTrue(cond: bool)The example for this item is shown with the ones above: testing-Assert.salam
AssertFalse
func AssertFalse(cond: bool)The example for this item is shown with the ones above: testing-Assert.salam
AssertEqInt
func AssertEqInt(got: int, want: int)The example for this item is shown with the ones above: testing-Assert.salam
AssertEqInt64
func AssertEqInt64(got: i64, want: i64)The example for this item is shown with the ones above: testing-Assert.salam
AssertEqFloat
func AssertEqFloat(got: f64, want: f64, eps: f64)eps. Never compare decimals with ==.The example for this item is shown with the ones above: testing-Assert.salam
AssertContains
func AssertContains(hay: str, needle: str)The example for this item is shown with the ones above: testing-Assert.salam
AssertEqStr
func AssertEqStr(got: str, want: str)The example for this item is shown with the ones above: testing-Assert.salam
AssertEqBool
func AssertEqBool(got: bool, want: bool)The example for this item is shown with the ones above: testing-Assert.salam
AssertNeInt
func AssertNeInt(a: int, b: int)The example for this item is shown with the ones above: testing-Assert.salam
AssertNeStr
func AssertNeStr(a: str, b: str)The example for this item is shown with the ones above: testing-Assert.salam
AssertNil
func AssertNil(p: void*)The example for this item is shown with the ones above: testing-Assert.salam
AssertNotNil
func AssertNotNil(p: void*)The example for this item is shown with the ones above: testing-Assert.salam
Reset
func Reset()The example for this item is shown with the ones above: testing-Assert.salam
Passed
func Passed(): intThe example for this item is shown with the ones above: testing-Assert.salam
Failed
func Failed(): intThe example for this item is shown with the ones above: testing-Assert.salam
Summary
func Summary(): int-- and tests: N passed, M failed to standard output and returns the number of failures.The example for this item is shown with the ones above: testing-Assert.salam
SetFormat
func SetFormat(format: str)Begin, End and Finish report: "teamcity" prints TeamCity service messages, which JetBrains IDEs and CI servers read as a test tree; anything else prints plain lines. salam test calls it for you.import testing
func add(a: int, b: int): int:
ret a + b
end
func main:
testing.SetFormat("human")
testing.SetFilter("add")
testing.SuiteStart("testing-Runner.salam")
if testing.Selected("test_add"):
testing.Begin("test_add", 12)
testing.AssertEqInt(add(2, 2), 4)
testing.End()
end
if testing.Selected("test_add_zero"):
testing.Begin("test_add_zero", 17)
testing.AssertEqInt(add(5, 0), 5)
testing.End()
end
testing.Finish()
endSetFilter
func SetFilter(part: str)salam test --run=... sets it.The example for this item is shown with the ones above: testing-Runner.salam
Selected
func Selected(name: str): boolThe example for this item is shown with the ones above: testing-Runner.salam
SuiteStart
func SuiteStart(file: str)salam test calls it for you.The example for this item is shown with the ones above: testing-Runner.salam
Begin
func Begin(name: str, line: int)End belong to it.The example for this item is shown with the ones above: testing-Runner.salam
End
func End()The example for this item is shown with the ones above: testing-Runner.salam
Finish
func Finish()The example for this item is shown with the ones above: testing-Runner.salam