Standard library · Encoding and data

data

Reading data for machine learning: CSV tables, batching samples, MNIST and images, and synthetic data.

import data

This package readies data for training a model. LoadCsv reads a CSV file into a Table, decides for each column whether it holds integers, floats, categories or free text, and turns categories into numbers by itself; split_xy then divides the table into a feature matrix and a label. MatrixDataset hands the rows out in shuffled mini-batches, LoadMnist, LoadIdx and LoadImageFolder read image sets, and the Make... functions build synthetic data from a fixed seed, for trying algorithms out without a file.

Everything comes out as a matrix or a tensor, ready for ml and nn. To read a CSV file cell by cell as text, use encoding.csv; for spreadsheets, excel; for arrays saved by NumPy, npy.

What trips people up

  • Category codes depend on the file. Codes are given in order of first appearance, so red may be 0 in the training file and 2 in the test file, and a column with 25 distinct values may be a category in one file and text in another. Load all rows from one file and split them, as the test-set guide does, or map new files through the training table's cat_levels.
  • One bad cell changes a column's kind. A single 12,5 or n/a in a number column turns the whole column into categories or text. Check kinds after loading.
  • Empty cells are NaN in number columns but their own category, the empty string, in category columns.
  • Only commas. The separator is always ,; convert semicolon or tab files first, for example with encoding.csv.

The guide below follows the order of a small machine learning project: load a CSV file and see how its columns were read, split it into features and a label and fit a first model, train in mini-batches, hold back a test set, and read the MNIST file format.

How LoadCsv reads each column

LoadCsv looks at every value of a column and picks its kind on its own. All whole numbers give COL_INT, all numbers COL_FLOAT. Text with a limited number of different values (at most 20, or a tenth of the rows when that is more, and fewer than the rows) becomes COL_CAT: each value gets a code from 0 in order of first appearance and cat_levels gives the names back. Text where every row is different, such as a free comment, stays COL_STR. An empty cell in a number column is stored as NaN, which math.IsNaN detects; in a category column it becomes a level of its own.

data-columns.salam
import data
import matrix
import math

func kind_name(k: int): str:
    if k == data.COL_INT: ret "int" end
    if k == data.COL_FLOAT: ret "float" end
    if k == data.COL_CAT: ret "category" end
    ret "text"
end

func main(): int throws:
    text := "id,city,age,income,comment\n1,Tehran,34,2.5,fast delivery\n2,Shiraz,,3.1,ok\n3,Tehran,51,4.0,late\n4,Tabriz,28,1.9,good price\n"
    mut t := try data.LoadCsv(text, true)
    each j, name in t.names:
        println name, "->", kind_name(t.kinds.Get(j))
    end
    levels := t.CatLevels("city")
    println "city levels:", levels
    city := t.ColIndex("city")
    age := t.ColIndex("age")
    repeat t.rows() in i:
        code := t.cells.at(i, city) as int
        mut shown := "missing"
        if math.IsNaN(t.cells.at(i, age)) == false: shown = "known" end
        println "row", i, "city code", code, "=", levels.Get(code), "age", shown
    end
    mut m := t.ToMatrix()
    println "to_matrix keeps", m.cols, "of", t.cols(), "columns"
    mut onehot := matrix.Zeros(t.rows(), levels.Len())
    repeat t.rows() in i: onehot.Set(i, t.cells.at(i, city) as int, 1.0) end
    println "one-hot row 3:", onehot.at(3, 0), onehot.at(3, 1), onehot.at(3, 2)
    matrix.Free(onehot)
    matrix.Free(m)
    t.Free()
    ret 0
end
Outputid -> int city -> category age -> int income -> float comment -> text city levels: ["Tehran", "Shiraz", "Tabriz"] row 0 city code 0 = Tehran age known row 1 city code 1 = Shiraz age missing row 2 city code 0 = Tehran age known row 3 city code 2 = Tabriz age known to_matrix keeps 4 of 5 columns one-hot row 3: 0 0 1

to_matrix and split_xy leave COL_STR columns out, which is why five columns become four. A category is stored as its code, which suits decision trees; for a linear model or a neural network build one-hot columns from the code, as the last lines do.

From a CSV file to a first model

split_xy takes the name of the label column and fills two matrices: every other usable column as features, and the label as a one column matrix. It returns false when the label is missing or is plain text. That is exactly the shape matrix functions expect, so a least squares fit with matrix.LinearRegression is one more line. Its result has the intercept first, then one coefficient per feature.

data-regression.salam
import data
import matrix
import conv

func main(): int throws:
    text := "area,rooms,price\n50,1,105\n70,2,147\n80,2,164\n100,3,208\n120,3,244\n150,4,305\n"
    mut t := try data.LoadCsv(text, true)
    mut x := matrix.Matrix {}
    mut y := matrix.Matrix {}
    if t.SplitXy("price", x, y) == false:
        println "no price column"
        ret 1
    end
    println "features:", x.rows, "x", x.cols, "label:", y.rows, "x", y.cols
    mut beta := matrix.LinearRegression(x, y)
    println "intercept:", conv.FormatFloatPrec(beta.at(0, 0), 2)
    println "per m2:", conv.FormatFloatPrec(beta.at(1, 0), 2)
    println "per room:", conv.FormatFloatPrec(beta.at(2, 0), 2)
    guess := beta.at(0, 0) + 90.0 * beta.at(1, 0) + 2.0 * beta.at(2, 0)
    println "90 m2 with 2 rooms:", conv.FormatFloatPrec(guess, 1)
    matrix.Free(beta)
    matrix.Free(x)
    matrix.Free(y)
    t.Free()
    ret 0
end
Outputfeatures: 6 x 2 label: 6 x 1 intercept: 7.33 per m2: 1.81 per room: 6.44 90 m2 with 2 rooms: 183.2

Training in mini-batches

Gradient descent on a large data set looks at a few rows at a time. Wrap the features and labels in a MatrixDataset, ask it for batches(size, shuffle, seed), and call next until it returns false; each call replaces the two matrices you pass with the next batch. reset(epoch) starts a new pass and shuffles again with the seed plus the epoch, so every epoch sees a different order but a run can be repeated exactly. MakeRegression builds a test problem with known true coefficients, so you can see that training found them.

data-minibatch.salam
import data
import matrix
import conv

func main:
    mut y := matrix.Matrix {}
    mut coef := matrix.Matrix {}
    x := data.MakeRegression(200, 3, 0.05, 11, y, coef)
    mut set := data.MatrixDataset { x = x, y = y }
    mut w := matrix.Zeros(3, 1)
    mut it := set.batches(20, true, 7)
    mut batch_x := matrix.Matrix {}
    mut batch_y := matrix.Matrix {}
    repeat 5 in epoch:
        it.reset(epoch as i64)
        mut loss := 0.0
        until it.Next(batch_x, batch_y):
            mut pred := matrix.MatMul(batch_x, w)
            mut diff := matrix.Sub(pred, batch_y)
            mut grad := matrix.TransposeMul(batch_x, diff)
            matrix.AddScaled(w, grad, -0.1 / (batch_x.rows as f64))
            loss += matrix.Dot(diff, diff)
            matrix.Free(pred)
            matrix.Free(diff)
            matrix.Free(grad)
        end
        println "epoch", epoch + 1, "batches", it.batches_per_epoch(), "loss", conv.FormatFloatPrec(loss / (set.Len() as f64), 4)
    end
    repeat 3 in j:
        println "w" + j, "learned", conv.FormatFloatPrec(w.at(j, 0), 2), "true", conv.FormatFloatPrec(coef.at(j, 0), 2)
    end
    matrix.Free(batch_x)
    matrix.Free(batch_y)
    matrix.Free(w)
    matrix.Free(coef)
    it.Free()
    set.Free()
    ret 0
end
Outputepoch 1 batches 10 loss 0.5009 epoch 2 batches 10 loss 0.0808 epoch 3 batches 10 loss 0.0147 epoch 4 batches 10 loss 0.0041 epoch 5 batches 10 loss 0.0024 w0 learned -0.86 true -0.88 w1 learned -0.54 true -0.54 w2 learned -0.26 true -0.25

next frees the matrices it is given before it fills them, so pass the same two variables every time and free them once after the loop.

Keeping a test set aside

A model has to be judged on rows it never saw. The simplest split reuses the batch iterator: shuffle once with a fixed seed, take a first batch the size of the training set and a second batch with the rest. Here a nearest-centre classifier learns the centre of each of three clusters from MakeBlobs and is scored on the 30 held back points.

data-split.salam
import data
import matrix

func nearest(centers: matrix.Matrix, x: matrix.Matrix, i: int): int:
    mut best := 0
    mut best_d := -1.0
    repeat centers.rows in c:
        mut d := 0.0
        repeat x.cols in j:
            diff := x.at(i, j) - centers.at(c, j)
            d += diff * diff
        end
        if best_d < 0.0 or d < best_d:
            best = c
            best_d = d
        end
    end
    ret best
end

func main:
    mut y := matrix.Matrix {}
    x := data.MakeBlobs(120, 3, 2, 1.0, 5, y)
    mut set := data.MatrixDataset { x = x, y = y }
    mut it := set.batches(90, true, 1)
    mut train_x := matrix.Matrix {}
    mut train_y := matrix.Matrix {}
    mut test_x := matrix.Matrix {}
    mut test_y := matrix.Matrix {}
    it.Next(train_x, train_y)
    it.Next(test_x, test_y)
    println "train:", train_x.rows, "test:", test_x.rows
    mut centers := matrix.Zeros(3, 2)
    mut sizes := matrix.Zeros(3, 1)
    repeat train_x.rows in i:
        c := train_y.at(i, 0) as int
        repeat 2 in j: centers.Set(c, j, centers.at(c, j) + train_x.at(i, j)) end
        sizes.Set(c, 0, sizes.at(c, 0) + 1.0)
    end
    repeat 3 in c:
        repeat 2 in j: centers.Set(c, j, centers.at(c, j) / sizes.at(c, 0)) end
    end
    mut right := 0
    repeat test_x.rows in i:
        if nearest(centers, test_x, i) == test_y.at(i, 0) as int: right++ end
    end
    println "correct on unseen points:", right, "of", test_x.rows
    each m in [train_x, train_y, test_x, test_y, centers, sizes]:
        mut t := m
        matrix.Free(t)
    end
    it.Free()
    set.Free()
    ret 0
end
Outputtrain: 90 test: 30 correct on unseen points: 25 of 30

Changing the seed gives another split and slightly different numbers. For a fair comparison of two models, use the same seed for both.

The MNIST file format

MNIST and many image sets that copy it use the IDX format: a magic number whose third byte is the element type and fourth the number of dimensions, the size of each dimension as a big-endian 32 bit number, then the raw bytes. LoadMnist reads an image file and a label file into a matrix with one image per row, pixels scaled from 0..255 to 0..1, and a one column label matrix. LoadIdx reads any IDX file as it is. The program writes three tiny 2x2 images itself so it runs without a download; for the real set, point the same call at train-images-idx3-ubyte and train-labels-idx1-ubyte.

data-mnist-format.salam
import data
import matrix
import compress
import archive
import encoding.binary
import conv

func write_idx(path: str, dims: Vector<int>, values: Vector<int>): bool:
    mut buf := compress.NewByteBuf()
    binary.WriteU8(buf, 0)
    binary.WriteU8(buf, 0)
    binary.WriteU8(buf, 8)
    binary.WriteU8(buf, dims.Len())
    each d in dims: binary.WriteU32BE(buf, d) end
    each v in values: binary.WriteU8(buf, v) end
    ok := archive.WriteBinaryFileBuf(path, buf)
    compress.BufFree(buf)
    ret ok
end

func main(): int throws:
    mut dims := Vector {} as Vector<int>
    dims.Push(3)
    dims.Push(2)
    dims.Push(2)
    mut pixels := Vector {} as Vector<int>
    repeat 12 in i: pixels.Push(i * 20) end
    write_idx("images.idx", dims, pixels)
    mut count := Vector {} as Vector<int>
    count.Push(3)
    mut labels := Vector {} as Vector<int>
    labels.Push(7)
    labels.Push(2)
    labels.Push(7)
    write_idx("labels.idx", count, labels)
    mut x := matrix.Matrix {}
    mut y := matrix.Matrix {}
    try data.LoadMnist("images.idx", "labels.idx", x, y)
    println "images:", x.rows, "pixels each:", x.cols, "labels:", y.rows
    repeat x.rows in i:
        println "label", y.at(i, 0), "pixels", conv.FormatFloatPrec(x.at(i, 0), 3), conv.FormatFloatPrec(x.at(i, 3), 3)
    end
    raw := try data.LoadIdx("images.idx")
    println "raw value of the last pixel:", raw.at(2, 3)
    try:
        data.LoadMnist("images.idx", "missing.idx", x, y)
    catch e:
        println "without labels:", e.ToStr()
    end
    matrix.Free(x)
    matrix.Free(y)
    ret 0
end
Outputimages: 3 pixels each: 4 labels: 3 label 7 pixels 0.000 0.235 label 2 pixels 0.314 0.549 label 7 pixels 0.627 0.863 raw value of the last pixel: 220 without labels: idx: cannot open file at missing.idx

Both readers are throws: a missing file is NotFound with the file in path, a file that is not IDX is idx.bad_magic, a file cut short is UnexpectedEnd with the byte offset, and images and labels of different lengths are idx.mismatch. You never get half filled matrices.

Types

BatchIter

struct BatchIter
A batch iterator.
NameTypeDescription
xmatrix.MatrixThe features.
ymatrix.MatrixThe labels.
batchint = 32The batch size.
shufflebool = trueShuffle the order.
seedi64 = 0The shuffle seed.
orderVectorThe order of the samples in this epoch.
posint = 0The current position.
data-Batch.salam
import data
import matrix
import ai.tensor

func main throws:
    mut y := matrix.Matrix {}
    mut x := data.MakeBlobs(10, 2, 3, 0.2, 1, y)
    mut set := data.MatrixDataset { x = x, y = y }
    println set.Len()
    mut it := set.batches(4, false, 0)
    println it.batches_per_epoch()
    mut batch_x := matrix.Matrix {}
    mut batch_y := matrix.Matrix {}
    until it.Next(batch_x, batch_y):
        println batch_x.rows, batch_x.cols, batch_y.rows
    end
    it.reset(1)
    mut t := tensor.Tensor {}
    mut labels := Vector {} as Vector<int>
    println it.next_tensor(t, labels), labels.Len()
    println data.YLabels(y).Len(), data.XTensor(x).shape
    try:
        data.ParseIdx(null, 0)
    catch e:
        println e.code
    end
    try:
        data.LoadIdx("missing.idx")
    catch e:
        println e.kind_name()
    end
    try:
        data.LoadMnist("a", "b", batch_x, batch_y)
    catch e:
        println e.path
    end
    mut classes := Vector {} as Vector<str>
    println data.LoadImageFolder("missing-folder", 8, 8, true, t, labels, classes)
    it.Free()
    set.Free()
end
Output10 3 4 3 4 4 3 4 2 3 2 true 4 10 [10, 3] binary.unexpected_end not found a false

Reset

func Reset(epoch: i64)

reset

func reset(epoch: i64)
Goes back to the start and builds a fresh order from the seed plus the epoch number.

The example for this item is shown with the ones above: data-Batch.salam

Next

func Next(x_out &: matrix.Matrix, y_out &: matrix.Matrix): bool

next

func next(x_out &: matrix.Matrix, y_out &: matrix.Matrix): bool
Puts the next batch in x_out and y_out; the last batch may be smaller. False when no batch is left.

The example for this item is shown with the ones above: data-Batch.salam

NextTensor

func NextTensor(x_out &: tensor.Tensor, labels_out &: Vector<int>): bool

next_tensor

func next_tensor(x_out &: tensor.Tensor, labels_out &: Vector<int>): bool
Hands out the next batch as a tensor and a list of integer labels.

The example for this item is shown with the ones above: data-Batch.salam

BatchesPerEpoch

func BatchesPerEpoch(): int

batches_per_epoch

func batches_per_epoch(): int
The number of batches in one epoch.

The example for this item is shown with the ones above: data-Batch.salam

Free

func Free()

free

func free()
Frees the memory.

The example for this item is shown with the ones above: data-Batch.salam

MatrixDataset

struct MatrixDataset
A dataset of a feature matrix and labels.
NameTypeDescription
xmatrix.MatrixThe feature matrix, one row per sample.
ymatrix.MatrixThe labels, one row per sample.

The example for this item is shown with the ones above: data-Batch.salam

Len

func Len(): int

len

func len(): int
The number of samples.

The example for this item is shown with the ones above: data-Batch.salam

Batches

func Batches(batch: int, shuffle: bool, seed: i64): BatchIter

batches

func batches(batch: int, shuffle: bool, seed: i64): BatchIter
An iterator that hands out the samples a batch at a time; when shuffle is true the order is shuffled from seed.

The example for this item is shown with the ones above: data-Batch.salam

Free

func Free()

free

func free()
Frees the memory.

The example for this item is shown with the ones above: data-Batch.salam

Table

struct Table
A data table with named columns, for machine learning.
NameTypeDescription
namesVectorThe column names.
kindsVectorThe kind of each column; one of the COL_ constants.
catsVector>The levels of each categorical column.
rawVector>The original text of the cells of COL_STR columns; empty for the other kinds.
cellsmatrix.MatrixThe numeric value of the cells; a categorical column holds the level number.
data-Table.salam
import data
import matrix

func main throws:
    text := "age,city,score,ok\n30,tehran,1.5,1\n25,shiraz,2.5,0\n41,tehran,3.0,1\n"
    mut t := try data.LoadCsv(text, true)
    println t.rows(), t.cols(), t.names, t.ColIndex("score"), t.ColIndex("x")
    println t.kinds[0] == data.COL_INT, t.kinds[1] == data.COL_CAT, t.kinds[2] == data.COL_FLOAT, data.COL_STR
    println t.CatLevels("city")
    mut m := t.ToMatrix()
    println m.rows, m.cols, m.at(2, 1)
    mut x := matrix.Matrix {}
    mut y := matrix.Matrix {}
    println t.SplitXy("ok", x, y), x.cols, y.at(1, 0)
    try:
        data.LoadCsvFile("missing.csv", true)
    catch e:
        println e.kind_name(), e.path
    end
    matrix.Free(m)
    matrix.Free(x)
    matrix.Free(y)
    t.Free()
end
Output3 4 ["age", "city", "score", "ok"] 2 -1 true true true 3 ["tehran", "shiraz"] 3 4 0 true 3 0 not found missing.csv

Rows

func Rows(): int

rows

func rows(): int
The number of data rows, not counting the header.

The example for this item is shown with the ones above: data-Table.salam

Cols

func Cols(): int

cols

func cols(): int
The number of columns.

The example for this item is shown with the ones above: data-Table.salam

ColIndex

func ColIndex(name: str): int

col_index

func col_index(name: str): int
The index of the column with this name; -1 when there is none.

The example for this item is shown with the ones above: data-Table.salam

CatLevels

func CatLevels(name: str): Vector<str>

cat_levels

func cat_levels(name: str): Vector<str>
The levels of a categorical column, in the order they were seen.

The example for this item is shown with the ones above: data-Table.salam

ToMatrix

func ToMatrix(): matrix.Matrix

to_matrix

func to_matrix(): matrix.Matrix
Turns every column into a numeric matrix.

The example for this item is shown with the ones above: data-Table.salam

SplitXY

func SplitXY(label: str, x &: matrix.Matrix, y &: matrix.Matrix): bool

SplitXy

func SplitXy(label: str, x &: matrix.Matrix, y &: matrix.Matrix): bool

split_xy

func split_xy(label: str, x &: matrix.Matrix, y &: matrix.Matrix): bool
Moves the label column into y and the rest into x.

The example for this item is shown with the ones above: data-Table.salam

Free

func Free()

free

func free()
Frees the memory.

The example for this item is shown with the ones above: data-Table.salam

Constants and variables

Column kinds COL_…

The kind of each table column, worked out as the CSV is read.
NameValue
COL_FLOAT0
COL_INT1
COL_CAT2
COL_STR3

The example for this item is shown with the ones above: data-Table.salam

Functions

XTensor

func XTensor(x &: matrix.Matrix): tensor.Tensor
Turns the feature matrix into a tensor.

The example for this item is shown with the ones above: data-Batch.salam

YLabels

func YLabels(y &: matrix.Matrix): Vector<int>
Turns the label column into a list of integers.

The example for this item is shown with the ones above: data-Batch.salam

LoadImageFolder

func LoadImageFolder(root: str, width: int, height: int, gray: bool, x &: tensor.Tensor, labels &: Vector<int>, classes &: Vector<str>): bool
Loads the images of a folder: each subfolder is one class. The images are resized to the given size, and turned grey when gray is true.

The example for this item is shown with the ones above: data-Batch.salam

ParseIdx

func ParseIdx(p: void*, n: int): Throws<matrix.Matrix>
Reads IDX format data, the MNIST format, from memory into a matrix. Throws on a bad header, an unknown element type and data that ends early.

The example for this item is shown with the ones above: data-Batch.salam

LoadIdx

func LoadIdx(path: str): Throws<matrix.Matrix>
Reads an IDX file into a matrix; a gzip compressed file (.gz) is unpacked first. Throws NotFound for a missing file and the errors of ParseIdx otherwise.

The example for this item is shown with the ones above: data-Batch.salam

LoadMnist

func LoadMnist(images_path: str, labels_path: str, x &: matrix.Matrix, y &: matrix.Matrix): Throws<bool>
Reads the MNIST image and label files; each image is one row of 784 values between zero and 1. Throws when either file cannot be read or their row counts differ; x and y are only replaced on success.

The example for this item is shown with the ones above: data-Batch.salam

LoadCsv

func LoadCsv(text: str, header: bool): Throws<Table>
Reads comma-separated text into a table and works out the kind of each column: integer, float, category or free text. With header false the columns are named col0, col1 and so on. Throws the encoding.csv errors, with line and column, on text it cannot read.

The example for this item is shown with the ones above: data-Table.salam

LoadCsvFile

func LoadCsvFile(path: str, header: bool): Throws<Table>
Reads a CSV file into a table. Throws NotFound for a missing file and the errors of LoadCsv otherwise, with the file name in path.

The example for this item is shown with the ones above: data-Table.salam

MakeBlobs

func MakeBlobs(n: int, centers: int, p: int, spread: f64, seed: i64, y &: matrix.Matrix): matrix.Matrix
Random samples in several clusters with random centres; the label of a sample is the number of its cluster.
data-Synth.salam
import data
import matrix

func count_ones(y: matrix.Matrix): int:
    mut n := 0
    repeat y.rows in i:
        if y.at(i, 0) == 1.0:
            n++
        end
    end
    ret n
end

func main:
    mut y := matrix.Matrix {}
    blobs := data.MakeBlobs(90, 3, 2, 0.5, 42, y)
    println blobs.rows, blobs.cols, y.rows
    moons := data.MakeMoons(100, 0.1, 7, y)
    println moons.rows, moons.cols, count_ones(y)
    circles := data.MakeCircles(60, 0.05, 0.5, 7, y)
    println circles.rows, count_ones(y)
    mut coef := matrix.Matrix {}
    reg := data.MakeRegression(50, 3, 0.1, 1, y, coef)
    println reg.rows, reg.cols, coef.rows
    cls := data.MakeClassification(40, 4, 2, 1.0, 3, y)
    println cls.rows, cls.cols, count_ones(y) > 0
end
Output90 2 90 100 2 50 60 30 50 3 3 40 4 true

MakeMoons

func MakeMoons(n: int, noise: f64, seed: i64, y &: matrix.Matrix): matrix.Matrix
Two interleaving crescents with two labels; for trying non linear separators out.

The example for this item is shown with the ones above: data-Synth.salam

MakeCircles

func MakeCircles(n: int, noise: f64, factor: f64, seed: i64, y &: matrix.Matrix): matrix.Matrix
Two concentric circles with two labels; factor is the radius ratio of the inner circle.

The example for this item is shown with the ones above: data-Synth.salam

MakeRegression

func MakeRegression(n: int, p: int, noise: f64, seed: i64, y &: matrix.Matrix, true_coef &: matrix.Matrix): matrix.Matrix
Linear regression data with random coefficients; the real coefficients come back in true_coef.

The example for this item is shown with the ones above: data-Synth.salam

MakeClassification

func MakeClassification(n: int, p: int, classes: int, sep: f64, seed: i64, y &: matrix.Matrix): matrix.Matrix
Classification data with several classes; sep sets how far apart the classes are.

The example for this item is shown with the ones above: data-Synth.salam