npy
Reading and writing NumPy .npy and .npz files.
import npy
.npy files are the standard way NumPy arrays are stored in Python, and .npz keeps several named arrays in one zip archive. With this package you can move data and model weights between Python and Salam.
An NpyArray is three things: shape, the size of each dimension; data, every element as an f64 in row-major (C) order; and dtype, the element type the file declared. The guide below goes from the usual case, moving a weight matrix between NumPy and Salam, to checkpoints with several arrays, arrays with more than two dimensions, the file layout itself and what happens with bad files.
Use it at the border with Python: training data prepared with NumPy or pandas, weights from a model trained in PyTorch and exported with NumPy, or results you want to plot in Python. Inside Salam, arrays live in a matrix or a tensor; for model files in other formats see onnx and gguf. For tables that people open in a spreadsheet, CSV with encoding.csv is friendlier.
What is supported
| Read | Write | |
|---|---|---|
| Element types | f8, f4, i1 to i8, u1 to u4, b1, little- or big-endian | <f8 only |
| Not read | u8 (uint64), f2 (float16), complex, strings and object arrays: an Unsupported error (npy.dtype) | |
| Memory order | C order; Fortran order for one or two dimensions | C order |
.npz | both numpy.savez and numpy.savez_compressed | uncompressed, like savez |
Every element becomes an f64, so int64 values above 253 (9,007,199,254,740,992) lose their last digits.
Save a matrix for NumPy, and load it back
FromMatrix turns a matrix into an NpyArray and WriteNpyFile writes it. In Python, numpy.load("weights.npy") opens the file as a 2 by 3 float64 array. The way back is ReadNpyFile followed by ToMatrix. Both report failure: WriteNpyFile returns false and ReadNpyFile throws, so the program runs it with try.
import npy
import matrix
func main(): int throws:
mut w := matrix.Zeros(2, 3)
repeat 2 in i:
repeat 3 in j:
w.Set(i, j, (i * 10 + j) as f64 / 4.0)
end
end
mut arr := npy.FromMatrix(w)
if npy.WriteNpyFile(arr, "weights.npy") == false:
println "could not write weights.npy"
ret 1
end
mut back := try npy.ReadNpyFile("weights.npy")
println "shape:", back.shape, "elements:", back.count()
mut m := try npy.ToMatrix(back)
repeat m.rows in i:
println m.at(i, 0), m.at(i, 1), m.at(i, 2)
end
println "same values:", matrix.Sum(m) == matrix.Sum(w)
matrix.Free(m)
matrix.Free(w)
back.Free()
arr.Free()
ret 0
endSeveral arrays in one .npz checkpoint
A trained model has more than one array: weights, biases, perhaps the mean and spread used to scale the inputs. NpzWrite stores them in one .npz file under the names you give in keys, the same file numpy.savez writes and numpy.load returns as a dictionary. NpzList shows what a file holds and NpzRead takes one array out by name. Below, a tiny linear layer is saved, loaded again and used for a prediction.
import npy
import matrix
func row(a: f64, b: f64): matrix.Matrix:
mut m := matrix.Zeros(1, 2)
m.Set(0, 0, a)
m.Set(0, 1, b)
ret m
end
func load(path: str, key: str): matrix.Matrix throws:
mut arr := try npy.NpzRead(path, key)
defer arr.Free()
ret try npy.ToMatrix(arr)
end
func main(): int throws:
mut w := matrix.Zeros(2, 2)
w.Set(0, 0, 0.5)
w.Set(0, 1, -1.0)
w.Set(1, 0, 2.0)
w.Set(1, 1, 0.25)
mut b := row(1.0, 0.0)
mut keys := Vector {} as Vector<str>
keys.Push("W")
keys.Push("b")
mut arrays := Vector {} as Vector<npy.NpyArray>
arrays.Push(npy.FromMatrix(w))
arrays.Push(npy.FromMatrix(b))
println "saved:", npy.NpzWrite("model.npz", keys, arrays)
println "arrays in file:", try npy.NpzList("model.npz")
mut lw := try load("model.npz", "W")
mut lb := try load("model.npz", "b")
mut x := row(3.0, 4.0)
mut xw := matrix.MatMul(x, lw)
mut y := matrix.AddRowVector(xw, lb)
println "prediction:", y.at(0, 0), y.at(0, 1)
try:
load("model.npz", "gamma")
catch e:
println "gamma:", e.ToStr()
end
repeat arrays.Len() in i:
mut a := arrays.Get(i)
a.Free()
end
matrix.Free(w)
matrix.Free(b)
matrix.Free(lw)
matrix.Free(lb)
matrix.Free(x)
matrix.Free(xw)
matrix.Free(y)
ret 0
endAsking for a name that is not in the file (gamma here) throws a NotFound error with the code npy.missing_key, so a loader can catch it and fall back to a default instead of crashing on an older checkpoint.
Arrays with more than two dimensions
A .npy file can hold any number of dimensions, such as a batch of images as (count, height, width). The package reads and writes them all, but ToMatrix only accepts one or two dimensions (a 1-D array becomes a single row) and throws an InvalidArgument error otherwise. For higher dimensions index data yourself: the element at [i][j][k] of shape (A, B, C) is at (i * B + j) * C + k.
import npy
import matrix
func at3(a: npy.NpyArray, i: int, j: int, k: int): f64:
rows := a.shape.Get(1)
cols := a.shape.Get(2)
ret a.data.Get((i * rows + j) * cols + k)
end
func main(): int throws:
mut cube := npy.NpyArray {}
cube.shape.Push(2)
cube.shape.Push(3)
cube.shape.Push(4)
repeat 24 in n: cube.data.Push(n as f64) end
println "ndim:", cube.ndim(), "count:", cube.count()
println "cube[1][2][3] =", at3(cube, 1, 2, 3)
try:
npy.ToMatrix(cube)
catch e:
println "3-D to matrix:", e.ToStr()
end
npy.WriteNpyFile(cube, "cube.npy")
mut back := try npy.ReadNpyFile("cube.npy")
println "read back:", back.shape, at3(back, 1, 2, 3)
mut line := npy.NpyArray {}
line.shape.Push(4)
repeat 4 in n: line.data.Push((n * n) as f64) end
mut m := try npy.ToMatrix(line)
println "1-D to matrix:", m.rows, "x", m.cols, m.at(0, 3)
matrix.Free(m)
line.Free()
back.Free()
cube.Free()
ret 0
endWhat is inside a .npy file
The format is simple enough to check by hand: the magic bytes \x93NUMPY, a version, a two byte header length, then a header that is a Python dictionary in plain text, padded with spaces, and finally the raw elements. WriteNpy gives you the bytes in a ByteBuf instead of a file, which is handy for sending an array over the network or storing it in a database.
import npy
import compress
import str
func main(): int throws:
mut arr := npy.NpyArray {}
arr.shape.Push(3)
repeat 3 in i: arr.data.Push((i + 1) as f64) end
mut buf := npy.WriteNpy(arr)
println "file size:", compress.BufLen(buf), "bytes"
println "magic byte:", compress.BufGet(buf, 0)
hlen := compress.BufGet(buf, 8) + 256 * compress.BufGet(buf, 9)
println "version:", compress.BufGet(buf, 6), "header length:", hlen
mut sb := str.NewBuilder()
repeat hlen in i:
str.BufAppendChar(sb, compress.BufGet(buf, 10 + i))
end
text := str.BufStr(sb)
trimmed := str.TrimRight(text)
println "header:", trimmed
println "data bytes:", compress.BufLen(buf) - 10 - hlen
mut back := try npy.ReadNpy(buf.data, compress.BufLen(buf))
println back.shape, back.data, back.dtype == npy.NPY_F64
str.BufFree(sb)
compress.BufFree(buf)
back.Free()
arr.Free()
ret 0
endThe header shows a limit to keep in mind: the writer always stores '<f8', little-endian 64 bit floats, whatever dtype says. On reading, files with float32, integer or bool elements are accepted, dtype tells you what they were and data holds them converted to f64. If Python needs another type, convert after loading with arr.astype(numpy.float32).
Bad, short and missing files
Every reader is throws: you never get an empty or half filled array back. A missing file is NotFound, a file that is not .npy at all is npy.bad_magic, an unknown element type is npy.dtype, and a file cut off in the middle is UnexpectedEnd with the code npy.truncated and the byte offset. The file name is in the error's path.
import npy
import compress
import io
func try_read(path: str):
try:
mut a := npy.ReadNpyFile(path)
println path, "->", a.shape
a.Free()
catch e:
println path, "->", e.kind_name(), "|", e.ToStr()
end
end
func main:
mut good := npy.NpyArray {}
good.shape.Push(2)
good.shape.Push(2)
repeat 4 in i: good.data.Push(i as f64) end
npy.WriteNpyFile(good, "good.npy")
io.WriteFile("notes.npy", "this is plain text")
mut buf := npy.WriteNpy(good)
mut half := compress.NewByteBuf()
compress.BufPushBytes(half, buf.data, compress.BufLen(buf) - 8)
try:
npy.ReadNpy(half.data, compress.BufLen(half))
catch e:
println "cut short ->", e.ToStr(), e.code
end
try_read("good.npy")
try_read("notes.npy")
try_read("missing.npy")
try:
npy.NpzList("missing.npz")
catch e:
println "missing.npz ->", e.is(ErrorKind.NotFound), e.path
end
compress.BufFree(half)
compress.BufFree(buf)
good.Free()
ret 0
endAn NpyArray owns its two vectors, so call free on arrays you no longer need in a long running program, and matrix.Free on the matrices made from them.
Types
NpyArray
struct NpyArray| Name | Type | Description |
|---|---|---|
shape | Vector | The size of each dimension, such as [2, 3]. |
dtype | int | The data type in the file; one of the NPY_ constants. |
data | Vector | The elements in row order. |
import npy
import matrix
func main throws:
mut m := matrix.Matrix { rows = 2, cols = 3 }
repeat 6 in i:
m.a.Push(i as f64)
end
mut a := npy.FromMatrix(m)
println a.shape, a.ndim(), a.count(), a.dtype == npy.NPY_F64
bytes := npy.WriteNpy(a)
println bytes.len > 64
mut back := try npy.ReadNpy(bytes.data, bytes.len)
println back.shape, back.data
m2 := try npy.ToMatrix(back)
println m2.at(1, 2)
println npy.WriteNpyFile(a, "a.npy")
mut from_file := try npy.ReadNpyFile("a.npy")
println from_file.count()
mut keys := Vector {} as Vector<str>
keys.Push("x")
keys.Push("y")
mut arrays := Vector {} as Vector<npy.NpyArray>
arrays.Push(a)
arrays.Push(back)
println npy.NpzWrite("ab.npz", keys, arrays)
println try npy.NpzList("ab.npz")
mut y := try npy.NpzRead("ab.npz", "y")
println y.shape
try:
npy.NpzRead("ab.npz", "z")
catch e:
println e.code
end
println npy.NPY_F32, npy.NPY_I8, npy.NPY_I16, npy.NPY_I32, npy.NPY_I64, npy.NPY_U8, npy.NPY_U16, npy.NPY_U32, npy.NPY_BOOL
y.Free()
from_file.Free()
back.Free()
a.Free()
endNdim
func Ndim(): intndim
func ndim(): intThe example for this item is shown with the ones above: npy-Npy.salam
Count
func Count(): intcount
func count(): intThe example for this item is shown with the ones above: npy-Npy.salam
Free
func Free()free
func free()The example for this item is shown with the ones above: npy-Npy.salam
Constants and variables
Data types NPY_…
| Name | Value |
|---|---|
NPY_F64 | 0 |
NPY_F32 | 1 |
NPY_I8 | 2 |
NPY_I16 | 3 |
NPY_I32 | 4 |
NPY_I64 | 5 |
NPY_U8 | 6 |
NPY_U16 | 7 |
NPY_U32 | 8 |
NPY_BOOL | 9 |
The example for this item is shown with the ones above: npy-Npy.salam
Functions
ReadNpy
func ReadNpy(p: void*, n: int): Throws<NpyArray>.npy file from memory. Throws on bad magic bytes, an unsupported version or dtype, a broken header or data that ends early.The example for this item is shown with the ones above: npy-Npy.salam
ReadNpyFile
func ReadNpyFile(path: str): Throws<NpyArray>.npy file. Throws NotFound when the file cannot be opened and the errors of ReadNpy otherwise, with the file name in path.The example for this item is shown with the ones above: npy-Npy.salam
WriteNpy
func WriteNpy(a &: NpyArray): compress.ByteBuf.npy bytes.The example for this item is shown with the ones above: npy-Npy.salam
WriteNpyFile
func WriteNpyFile(a &: NpyArray, path: str): bool.npy file.The example for this item is shown with the ones above: npy-Npy.salam
ToMatrix
func ToMatrix(a &: NpyArray): Throws<matrix.Matrix>InvalidArgument for any other number of dimensions.The example for this item is shown with the ones above: npy-Npy.salam
FromMatrix
func FromMatrix(m &: matrix.Matrix): NpyArrayThe example for this item is shown with the ones above: npy-Npy.salam
NpzList
func NpzList(path: str): Throws<Vector<str>>.npz file. Throws when the file cannot be opened or is not a zip archive.The example for this item is shown with the ones above: npy-Npy.salam
NpzRead
func NpzRead(path: str, key: str): Throws<NpyArray>.npz; pass the name without .npy, as numpy.savez keys are. Compressed files from savez_compressed work too. Throws npy.missing_key when there is no such array.The example for this item is shown with the ones above: npy-Npy.salam
NpzWrite
func NpzWrite(path: str, keys &: Vector<str>, arrays &: Vector<NpyArray>): bool.npz file, uncompressed. keys and arrays go in the same order.The example for this item is shown with the ones above: npy-Npy.salam