Standard library · Encoding and data

npy

Reading and writing NumPy .npy and .npz files.

import npy

.npy files are the standard way NumPy arrays are stored in Python, and .npz keeps several named arrays in one zip archive. With this package you can move data and model weights between Python and Salam.

An NpyArray is three things: shape, the size of each dimension; data, every element as an f64 in row-major (C) order; and dtype, the element type the file declared. The guide below goes from the usual case, moving a weight matrix between NumPy and Salam, to checkpoints with several arrays, arrays with more than two dimensions, the file layout itself and what happens with bad files.

Use it at the border with Python: training data prepared with NumPy or pandas, weights from a model trained in PyTorch and exported with NumPy, or results you want to plot in Python. Inside Salam, arrays live in a matrix or a tensor; for model files in other formats see onnx and gguf. For tables that people open in a spreadsheet, CSV with encoding.csv is friendlier.

What is supported

ReadWrite
Element typesf8, f4, i1 to i8, u1 to u4, b1, little- or big-endian<f8 only
Not readu8 (uint64), f2 (float16), complex, strings and object arrays: an Unsupported error (npy.dtype)
Memory orderC order; Fortran order for one or two dimensionsC order
.npzboth numpy.savez and numpy.savez_compresseduncompressed, like savez

Every element becomes an f64, so int64 values above 253 (9,007,199,254,740,992) lose their last digits.

Save a matrix for NumPy, and load it back

FromMatrix turns a matrix into an NpyArray and WriteNpyFile writes it. In Python, numpy.load("weights.npy") opens the file as a 2 by 3 float64 array. The way back is ReadNpyFile followed by ToMatrix. Both report failure: WriteNpyFile returns false and ReadNpyFile throws, so the program runs it with try.

npy-weights.salam
import npy
import matrix

func main(): int throws:
    mut w := matrix.Zeros(2, 3)
    repeat 2 in i:
        repeat 3 in j:
            w.Set(i, j, (i * 10 + j) as f64 / 4.0)
        end
    end
    mut arr := npy.FromMatrix(w)
    if npy.WriteNpyFile(arr, "weights.npy") == false:
        println "could not write weights.npy"
        ret 1
    end
    mut back := try npy.ReadNpyFile("weights.npy")
    println "shape:", back.shape, "elements:", back.count()
    mut m := try npy.ToMatrix(back)
    repeat m.rows in i:
        println m.at(i, 0), m.at(i, 1), m.at(i, 2)
    end
    println "same values:", matrix.Sum(m) == matrix.Sum(w)
    matrix.Free(m)
    matrix.Free(w)
    back.Free()
    arr.Free()
    ret 0
end
Outputshape: [2, 3] elements: 6 0 0.25 0.5 2.5 2.75 3 same values: true

Several arrays in one .npz checkpoint

A trained model has more than one array: weights, biases, perhaps the mean and spread used to scale the inputs. NpzWrite stores them in one .npz file under the names you give in keys, the same file numpy.savez writes and numpy.load returns as a dictionary. NpzList shows what a file holds and NpzRead takes one array out by name. Below, a tiny linear layer is saved, loaded again and used for a prediction.

npy-checkpoint.salam
import npy
import matrix

func row(a: f64, b: f64): matrix.Matrix:
    mut m := matrix.Zeros(1, 2)
    m.Set(0, 0, a)
    m.Set(0, 1, b)
    ret m
end

func load(path: str, key: str): matrix.Matrix throws:
    mut arr := try npy.NpzRead(path, key)
    defer arr.Free()
    ret try npy.ToMatrix(arr)
end

func main(): int throws:
    mut w := matrix.Zeros(2, 2)
    w.Set(0, 0, 0.5)
    w.Set(0, 1, -1.0)
    w.Set(1, 0, 2.0)
    w.Set(1, 1, 0.25)
    mut b := row(1.0, 0.0)
    mut keys := Vector {} as Vector<str>
    keys.Push("W")
    keys.Push("b")
    mut arrays := Vector {} as Vector<npy.NpyArray>
    arrays.Push(npy.FromMatrix(w))
    arrays.Push(npy.FromMatrix(b))
    println "saved:", npy.NpzWrite("model.npz", keys, arrays)
    println "arrays in file:", try npy.NpzList("model.npz")
    mut lw := try load("model.npz", "W")
    mut lb := try load("model.npz", "b")
    mut x := row(3.0, 4.0)
    mut xw := matrix.MatMul(x, lw)
    mut y := matrix.AddRowVector(xw, lb)
    println "prediction:", y.at(0, 0), y.at(0, 1)
    try:
        load("model.npz", "gamma")
    catch e:
        println "gamma:", e.ToStr()
    end
    repeat arrays.Len() in i:
        mut a := arrays.Get(i)
        a.Free()
    end
    matrix.Free(w)
    matrix.Free(b)
    matrix.Free(lw)
    matrix.Free(lb)
    matrix.Free(x)
    matrix.Free(xw)
    matrix.Free(y)
    ret 0
end
Outputsaved: true arrays in file: ["W", "b"] prediction: 10.5 -2 gamma: npy: no array named 'gamma' at model.npz

Asking for a name that is not in the file (gamma here) throws a NotFound error with the code npy.missing_key, so a loader can catch it and fall back to a default instead of crashing on an older checkpoint.

Arrays with more than two dimensions

A .npy file can hold any number of dimensions, such as a batch of images as (count, height, width). The package reads and writes them all, but ToMatrix only accepts one or two dimensions (a 1-D array becomes a single row) and throws an InvalidArgument error otherwise. For higher dimensions index data yourself: the element at [i][j][k] of shape (A, B, C) is at (i * B + j) * C + k.

npy-shapes.salam
import npy
import matrix

func at3(a: npy.NpyArray, i: int, j: int, k: int): f64:
    rows := a.shape.Get(1)
    cols := a.shape.Get(2)
    ret a.data.Get((i * rows + j) * cols + k)
end

func main(): int throws:
    mut cube := npy.NpyArray {}
    cube.shape.Push(2)
    cube.shape.Push(3)
    cube.shape.Push(4)
    repeat 24 in n: cube.data.Push(n as f64) end
    println "ndim:", cube.ndim(), "count:", cube.count()
    println "cube[1][2][3] =", at3(cube, 1, 2, 3)
    try:
        npy.ToMatrix(cube)
    catch e:
        println "3-D to matrix:", e.ToStr()
    end
    npy.WriteNpyFile(cube, "cube.npy")
    mut back := try npy.ReadNpyFile("cube.npy")
    println "read back:", back.shape, at3(back, 1, 2, 3)
    mut line := npy.NpyArray {}
    line.shape.Push(4)
    repeat 4 in n: line.data.Push((n * n) as f64) end
    mut m := try npy.ToMatrix(line)
    println "1-D to matrix:", m.rows, "x", m.cols, m.at(0, 3)
    matrix.Free(m)
    line.Free()
    back.Free()
    cube.Free()
    ret 0
end
Outputndim: 3 count: 24 cube[1][2][3] = 23 3-D to matrix: npy: ToMatrix needs a 1-D or 2-D array, got 3 dimensions read back: [2, 3, 4] 23 1-D to matrix: 1 x 4 9

What is inside a .npy file

The format is simple enough to check by hand: the magic bytes \x93NUMPY, a version, a two byte header length, then a header that is a Python dictionary in plain text, padded with spaces, and finally the raw elements. WriteNpy gives you the bytes in a ByteBuf instead of a file, which is handy for sending an array over the network or storing it in a database.

npy-header.salam
import npy
import compress
import str

func main(): int throws:
    mut arr := npy.NpyArray {}
    arr.shape.Push(3)
    repeat 3 in i: arr.data.Push((i + 1) as f64) end
    mut buf := npy.WriteNpy(arr)
    println "file size:", compress.BufLen(buf), "bytes"
    println "magic byte:", compress.BufGet(buf, 0)
    hlen := compress.BufGet(buf, 8) + 256 * compress.BufGet(buf, 9)
    println "version:", compress.BufGet(buf, 6), "header length:", hlen
    mut sb := str.NewBuilder()
    repeat hlen in i:
        str.BufAppendChar(sb, compress.BufGet(buf, 10 + i))
    end
    text := str.BufStr(sb)
    trimmed := str.TrimRight(text)
    println "header:", trimmed
    println "data bytes:", compress.BufLen(buf) - 10 - hlen
    mut back := try npy.ReadNpy(buf.data, compress.BufLen(buf))
    println back.shape, back.data, back.dtype == npy.NPY_F64
    str.BufFree(sb)
    compress.BufFree(buf)
    back.Free()
    arr.Free()
    ret 0
end
Outputfile size: 152 bytes magic byte: 147 version: 1 header length: 118 header: {'descr': '<f8', 'fortran_order': False, 'shape': (3,), } data bytes: 24 [3] [1, 2, 3] true

The header shows a limit to keep in mind: the writer always stores '<f8', little-endian 64 bit floats, whatever dtype says. On reading, files with float32, integer or bool elements are accepted, dtype tells you what they were and data holds them converted to f64. If Python needs another type, convert after loading with arr.astype(numpy.float32).

Bad, short and missing files

Every reader is throws: you never get an empty or half filled array back. A missing file is NotFound, a file that is not .npy at all is npy.bad_magic, an unknown element type is npy.dtype, and a file cut off in the middle is UnexpectedEnd with the code npy.truncated and the byte offset. The file name is in the error's path.

npy-errors.salam
import npy
import compress
import io

func try_read(path: str):
    try:
        mut a := npy.ReadNpyFile(path)
        println path, "->", a.shape
        a.Free()
    catch e:
        println path, "->", e.kind_name(), "|", e.ToStr()
    end
end

func main:
    mut good := npy.NpyArray {}
    good.shape.Push(2)
    good.shape.Push(2)
    repeat 4 in i: good.data.Push(i as f64) end
    npy.WriteNpyFile(good, "good.npy")
    io.WriteFile("notes.npy", "this is plain text")
    mut buf := npy.WriteNpy(good)
    mut half := compress.NewByteBuf()
    compress.BufPushBytes(half, buf.data, compress.BufLen(buf) - 8)
    try:
        npy.ReadNpy(half.data, compress.BufLen(half))
    catch e:
        println "cut short ->", e.ToStr(), e.code
    end
    try_read("good.npy")
    try_read("notes.npy")
    try_read("missing.npy")
    try:
        npy.NpzList("missing.npz")
    catch e:
        println "missing.npz ->", e.is(ErrorKind.NotFound), e.path
    end
    compress.BufFree(half)
    compress.BufFree(buf)
    good.Free()
    ret 0
end
Outputcut short -> npy: the data ends early: 4 elements need 32 bytes, 24 are left (offset 128) npy.truncated good.npy -> [2, 2] notes.npy -> syntax error | npy: not a .npy file (bad magic bytes) at notes.npy (offset 0) missing.npy -> not found | npy: cannot open file at missing.npy missing.npz -> true missing.npz

An NpyArray owns its two vectors, so call free on arrays you no longer need in a long running program, and matrix.Free on the matrices made from them.

Types

NpyArray

struct NpyArray
One multi dimensional NumPy array. The data is always held in memory as 64 bit floats.
NameTypeDescription
shapeVectorThe size of each dimension, such as [2, 3].
dtypeintThe data type in the file; one of the NPY_ constants.
dataVectorThe elements in row order.
npy-Npy.salam
import npy
import matrix

func main throws:
    mut m := matrix.Matrix { rows = 2, cols = 3 }
    repeat 6 in i:
        m.a.Push(i as f64)
    end
    mut a := npy.FromMatrix(m)
    println a.shape, a.ndim(), a.count(), a.dtype == npy.NPY_F64
    bytes := npy.WriteNpy(a)
    println bytes.len > 64
    mut back := try npy.ReadNpy(bytes.data, bytes.len)
    println back.shape, back.data
    m2 := try npy.ToMatrix(back)
    println m2.at(1, 2)
    println npy.WriteNpyFile(a, "a.npy")
    mut from_file := try npy.ReadNpyFile("a.npy")
    println from_file.count()
    mut keys := Vector {} as Vector<str>
    keys.Push("x")
    keys.Push("y")
    mut arrays := Vector {} as Vector<npy.NpyArray>
    arrays.Push(a)
    arrays.Push(back)
    println npy.NpzWrite("ab.npz", keys, arrays)
    println try npy.NpzList("ab.npz")
    mut y := try npy.NpzRead("ab.npz", "y")
    println y.shape
    try:
        npy.NpzRead("ab.npz", "z")
    catch e:
        println e.code
    end
    println npy.NPY_F32, npy.NPY_I8, npy.NPY_I16, npy.NPY_I32, npy.NPY_I64, npy.NPY_U8, npy.NPY_U16, npy.NPY_U32, npy.NPY_BOOL
    y.Free()
    from_file.Free()
    back.Free()
    a.Free()
end
Output[2, 3] 2 6 true true [2, 3] [0, 1, 2, 3, 4, 5] 5 true 6 true ["x", "y"] [2, 3] npy.missing_key 1 2 3 4 5 6 7 8 9

Ndim

func Ndim(): int

ndim

func ndim(): int
The number of dimensions.

The example for this item is shown with the ones above: npy-Npy.salam

Count

func Count(): int

count

func count(): int
The total number of elements.

The example for this item is shown with the ones above: npy-Npy.salam

Free

func Free()

free

func free()
Frees the memory of the array.

The example for this item is shown with the ones above: npy-Npy.salam

Constants and variables

Data types NPY_…

The type of the elements in the file: 8 and 4 byte floats, 1 to 8 byte signed and unsigned integers, and boolean.
NameValue
NPY_F640
NPY_F321
NPY_I82
NPY_I163
NPY_I324
NPY_I645
NPY_U86
NPY_U167
NPY_U328
NPY_BOOL9

The example for this item is shown with the ones above: npy-Npy.salam

Functions

ReadNpy

func ReadNpy(p: void*, n: int): Throws<NpyArray>
Reads the bytes of a .npy file from memory. Throws on bad magic bytes, an unsupported version or dtype, a broken header or data that ends early.

The example for this item is shown with the ones above: npy-Npy.salam

ReadNpyFile

func ReadNpyFile(path: str): Throws<NpyArray>
Reads a .npy file. Throws NotFound when the file cannot be opened and the errors of ReadNpy otherwise, with the file name in path.

The example for this item is shown with the ones above: npy-Npy.salam

WriteNpy

func WriteNpy(a &: NpyArray): compress.ByteBuf
Turns the array into .npy bytes.

The example for this item is shown with the ones above: npy-Npy.salam

WriteNpyFile

func WriteNpyFile(a &: NpyArray, path: str): bool
Writes the array to a .npy file.

The example for this item is shown with the ones above: npy-Npy.salam

ToMatrix

func ToMatrix(a &: NpyArray): Throws<matrix.Matrix>
Turns a one or two dimensional array into a matrix (one dimension gives a single row); throws InvalidArgument for any other number of dimensions.

The example for this item is shown with the ones above: npy-Npy.salam

FromMatrix

func FromMatrix(m &: matrix.Matrix): NpyArray
Turns a matrix into a two dimensional array of 64 bit floats.

The example for this item is shown with the ones above: npy-Npy.salam

NpzList

func NpzList(path: str): Throws<Vector<str>>
The names of the arrays in a .npz file. Throws when the file cannot be opened or is not a zip archive.

The example for this item is shown with the ones above: npy-Npy.salam

NpzRead

func NpzRead(path: str, key: str): Throws<NpyArray>
Reads the array with the given name out of a .npz; pass the name without .npy, as numpy.savez keys are. Compressed files from savez_compressed work too. Throws npy.missing_key when there is no such array.

The example for this item is shown with the ones above: npy-Npy.salam

NpzWrite

func NpzWrite(path: str, keys &: Vector<str>, arrays &: Vector<NpyArray>): bool
Writes several arrays with their names into one .npz file, uncompressed. keys and arrays go in the same order.

The example for this item is shown with the ones above: npy-Npy.salam