data
Reading data for machine learning: CSV tables, batching samples, MNIST and images, and synthetic data.
import data
This package readies data for training a model. LoadCsv reads a CSV file into a Table, decides for each column whether it holds integers, floats, categories or free text, and turns categories into numbers by itself; split_xy then divides the table into a feature matrix and a label. MatrixDataset hands the rows out in shuffled mini-batches, LoadMnist, LoadIdx and LoadImageFolder read image sets, and the Make... functions build synthetic data from a fixed seed, for trying algorithms out without a file.
Everything comes out as a matrix or a tensor, ready for ml and nn. To read a CSV file cell by cell as text, use encoding.csv; for spreadsheets, excel; for arrays saved by NumPy, npy.
What trips people up
- Category codes depend on the file. Codes are given in order of first appearance, so
redmay be 0 in the training file and 2 in the test file, and a column with 25 distinct values may be a category in one file and text in another. Load all rows from one file and split them, as the test-set guide does, or map new files through the training table'scat_levels. - One bad cell changes a column's kind. A single
12,5orn/ain a number column turns the whole column into categories or text. Checkkindsafter loading. - Empty cells are NaN in number columns but their own category, the empty string, in category columns.
- Only commas. The separator is always
,; convert semicolon or tab files first, for example with encoding.csv.
The guide below follows the order of a small machine learning project: load a CSV file and see how its columns were read, split it into features and a label and fit a first model, train in mini-batches, hold back a test set, and read the MNIST file format.
How LoadCsv reads each column
LoadCsv looks at every value of a column and picks its kind on its own. All whole numbers give COL_INT, all numbers COL_FLOAT. Text with a limited number of different values (at most 20, or a tenth of the rows when that is more, and fewer than the rows) becomes COL_CAT: each value gets a code from 0 in order of first appearance and cat_levels gives the names back. Text where every row is different, such as a free comment, stays COL_STR. An empty cell in a number column is stored as NaN, which math.IsNaN detects; in a category column it becomes a level of its own.
import data
import matrix
import math
func kind_name(k: int): str:
if k == data.COL_INT: ret "int" end
if k == data.COL_FLOAT: ret "float" end
if k == data.COL_CAT: ret "category" end
ret "text"
end
func main(): int throws:
text := "id,city,age,income,comment\n1,Tehran,34,2.5,fast delivery\n2,Shiraz,,3.1,ok\n3,Tehran,51,4.0,late\n4,Tabriz,28,1.9,good price\n"
mut t := try data.LoadCsv(text, true)
each j, name in t.names:
println name, "->", kind_name(t.kinds.Get(j))
end
levels := t.CatLevels("city")
println "city levels:", levels
city := t.ColIndex("city")
age := t.ColIndex("age")
repeat t.rows() in i:
code := t.cells.at(i, city) as int
mut shown := "missing"
if math.IsNaN(t.cells.at(i, age)) == false: shown = "known" end
println "row", i, "city code", code, "=", levels.Get(code), "age", shown
end
mut m := t.ToMatrix()
println "to_matrix keeps", m.cols, "of", t.cols(), "columns"
mut onehot := matrix.Zeros(t.rows(), levels.Len())
repeat t.rows() in i: onehot.Set(i, t.cells.at(i, city) as int, 1.0) end
println "one-hot row 3:", onehot.at(3, 0), onehot.at(3, 1), onehot.at(3, 2)
matrix.Free(onehot)
matrix.Free(m)
t.Free()
ret 0
endto_matrix and split_xy leave COL_STR columns out, which is why five columns become four. A category is stored as its code, which suits decision trees; for a linear model or a neural network build one-hot columns from the code, as the last lines do.
From a CSV file to a first model
split_xy takes the name of the label column and fills two matrices: every other usable column as features, and the label as a one column matrix. It returns false when the label is missing or is plain text. That is exactly the shape matrix functions expect, so a least squares fit with matrix.LinearRegression is one more line. Its result has the intercept first, then one coefficient per feature.
import data
import matrix
import conv
func main(): int throws:
text := "area,rooms,price\n50,1,105\n70,2,147\n80,2,164\n100,3,208\n120,3,244\n150,4,305\n"
mut t := try data.LoadCsv(text, true)
mut x := matrix.Matrix {}
mut y := matrix.Matrix {}
if t.SplitXy("price", x, y) == false:
println "no price column"
ret 1
end
println "features:", x.rows, "x", x.cols, "label:", y.rows, "x", y.cols
mut beta := matrix.LinearRegression(x, y)
println "intercept:", conv.FormatFloatPrec(beta.at(0, 0), 2)
println "per m2:", conv.FormatFloatPrec(beta.at(1, 0), 2)
println "per room:", conv.FormatFloatPrec(beta.at(2, 0), 2)
guess := beta.at(0, 0) + 90.0 * beta.at(1, 0) + 2.0 * beta.at(2, 0)
println "90 m2 with 2 rooms:", conv.FormatFloatPrec(guess, 1)
matrix.Free(beta)
matrix.Free(x)
matrix.Free(y)
t.Free()
ret 0
endTraining in mini-batches
Gradient descent on a large data set looks at a few rows at a time. Wrap the features and labels in a MatrixDataset, ask it for batches(size, shuffle, seed), and call next until it returns false; each call replaces the two matrices you pass with the next batch. reset(epoch) starts a new pass and shuffles again with the seed plus the epoch, so every epoch sees a different order but a run can be repeated exactly. MakeRegression builds a test problem with known true coefficients, so you can see that training found them.
import data
import matrix
import conv
func main:
mut y := matrix.Matrix {}
mut coef := matrix.Matrix {}
x := data.MakeRegression(200, 3, 0.05, 11, y, coef)
mut set := data.MatrixDataset { x = x, y = y }
mut w := matrix.Zeros(3, 1)
mut it := set.batches(20, true, 7)
mut batch_x := matrix.Matrix {}
mut batch_y := matrix.Matrix {}
repeat 5 in epoch:
it.reset(epoch as i64)
mut loss := 0.0
until it.Next(batch_x, batch_y):
mut pred := matrix.MatMul(batch_x, w)
mut diff := matrix.Sub(pred, batch_y)
mut grad := matrix.TransposeMul(batch_x, diff)
matrix.AddScaled(w, grad, -0.1 / (batch_x.rows as f64))
loss += matrix.Dot(diff, diff)
matrix.Free(pred)
matrix.Free(diff)
matrix.Free(grad)
end
println "epoch", epoch + 1, "batches", it.batches_per_epoch(), "loss", conv.FormatFloatPrec(loss / (set.Len() as f64), 4)
end
repeat 3 in j:
println "w" + j, "learned", conv.FormatFloatPrec(w.at(j, 0), 2), "true", conv.FormatFloatPrec(coef.at(j, 0), 2)
end
matrix.Free(batch_x)
matrix.Free(batch_y)
matrix.Free(w)
matrix.Free(coef)
it.Free()
set.Free()
ret 0
endnext frees the matrices it is given before it fills them, so pass the same two variables every time and free them once after the loop.
Keeping a test set aside
A model has to be judged on rows it never saw. The simplest split reuses the batch iterator: shuffle once with a fixed seed, take a first batch the size of the training set and a second batch with the rest. Here a nearest-centre classifier learns the centre of each of three clusters from MakeBlobs and is scored on the 30 held back points.
import data
import matrix
func nearest(centers: matrix.Matrix, x: matrix.Matrix, i: int): int:
mut best := 0
mut best_d := -1.0
repeat centers.rows in c:
mut d := 0.0
repeat x.cols in j:
diff := x.at(i, j) - centers.at(c, j)
d += diff * diff
end
if best_d < 0.0 or d < best_d:
best = c
best_d = d
end
end
ret best
end
func main:
mut y := matrix.Matrix {}
x := data.MakeBlobs(120, 3, 2, 1.0, 5, y)
mut set := data.MatrixDataset { x = x, y = y }
mut it := set.batches(90, true, 1)
mut train_x := matrix.Matrix {}
mut train_y := matrix.Matrix {}
mut test_x := matrix.Matrix {}
mut test_y := matrix.Matrix {}
it.Next(train_x, train_y)
it.Next(test_x, test_y)
println "train:", train_x.rows, "test:", test_x.rows
mut centers := matrix.Zeros(3, 2)
mut sizes := matrix.Zeros(3, 1)
repeat train_x.rows in i:
c := train_y.at(i, 0) as int
repeat 2 in j: centers.Set(c, j, centers.at(c, j) + train_x.at(i, j)) end
sizes.Set(c, 0, sizes.at(c, 0) + 1.0)
end
repeat 3 in c:
repeat 2 in j: centers.Set(c, j, centers.at(c, j) / sizes.at(c, 0)) end
end
mut right := 0
repeat test_x.rows in i:
if nearest(centers, test_x, i) == test_y.at(i, 0) as int: right++ end
end
println "correct on unseen points:", right, "of", test_x.rows
each m in [train_x, train_y, test_x, test_y, centers, sizes]:
mut t := m
matrix.Free(t)
end
it.Free()
set.Free()
ret 0
endChanging the seed gives another split and slightly different numbers. For a fair comparison of two models, use the same seed for both.
The MNIST file format
MNIST and many image sets that copy it use the IDX format: a magic number whose third byte is the element type and fourth the number of dimensions, the size of each dimension as a big-endian 32 bit number, then the raw bytes. LoadMnist reads an image file and a label file into a matrix with one image per row, pixels scaled from 0..255 to 0..1, and a one column label matrix. LoadIdx reads any IDX file as it is. The program writes three tiny 2x2 images itself so it runs without a download; for the real set, point the same call at train-images-idx3-ubyte and train-labels-idx1-ubyte.
import data
import matrix
import compress
import archive
import encoding.binary
import conv
func write_idx(path: str, dims: Vector<int>, values: Vector<int>): bool:
mut buf := compress.NewByteBuf()
binary.WriteU8(buf, 0)
binary.WriteU8(buf, 0)
binary.WriteU8(buf, 8)
binary.WriteU8(buf, dims.Len())
each d in dims: binary.WriteU32BE(buf, d) end
each v in values: binary.WriteU8(buf, v) end
ok := archive.WriteBinaryFileBuf(path, buf)
compress.BufFree(buf)
ret ok
end
func main(): int throws:
mut dims := Vector {} as Vector<int>
dims.Push(3)
dims.Push(2)
dims.Push(2)
mut pixels := Vector {} as Vector<int>
repeat 12 in i: pixels.Push(i * 20) end
write_idx("images.idx", dims, pixels)
mut count := Vector {} as Vector<int>
count.Push(3)
mut labels := Vector {} as Vector<int>
labels.Push(7)
labels.Push(2)
labels.Push(7)
write_idx("labels.idx", count, labels)
mut x := matrix.Matrix {}
mut y := matrix.Matrix {}
try data.LoadMnist("images.idx", "labels.idx", x, y)
println "images:", x.rows, "pixels each:", x.cols, "labels:", y.rows
repeat x.rows in i:
println "label", y.at(i, 0), "pixels", conv.FormatFloatPrec(x.at(i, 0), 3), conv.FormatFloatPrec(x.at(i, 3), 3)
end
raw := try data.LoadIdx("images.idx")
println "raw value of the last pixel:", raw.at(2, 3)
try:
data.LoadMnist("images.idx", "missing.idx", x, y)
catch e:
println "without labels:", e.ToStr()
end
matrix.Free(x)
matrix.Free(y)
ret 0
endBoth readers are throws: a missing file is NotFound with the file in path, a file that is not IDX is idx.bad_magic, a file cut short is UnexpectedEnd with the byte offset, and images and labels of different lengths are idx.mismatch. You never get half filled matrices.
Types
BatchIter
struct BatchIter| Name | Type | Description |
|---|---|---|
x | matrix.Matrix | The features. |
y | matrix.Matrix | The labels. |
batch | int = 32 | The batch size. |
shuffle | bool = true | Shuffle the order. |
seed | i64 = 0 | The shuffle seed. |
order | Vector | The order of the samples in this epoch. |
pos | int = 0 | The current position. |
import data
import matrix
import ai.tensor
func main throws:
mut y := matrix.Matrix {}
mut x := data.MakeBlobs(10, 2, 3, 0.2, 1, y)
mut set := data.MatrixDataset { x = x, y = y }
println set.Len()
mut it := set.batches(4, false, 0)
println it.batches_per_epoch()
mut batch_x := matrix.Matrix {}
mut batch_y := matrix.Matrix {}
until it.Next(batch_x, batch_y):
println batch_x.rows, batch_x.cols, batch_y.rows
end
it.reset(1)
mut t := tensor.Tensor {}
mut labels := Vector {} as Vector<int>
println it.next_tensor(t, labels), labels.Len()
println data.YLabels(y).Len(), data.XTensor(x).shape
try:
data.ParseIdx(null, 0)
catch e:
println e.code
end
try:
data.LoadIdx("missing.idx")
catch e:
println e.kind_name()
end
try:
data.LoadMnist("a", "b", batch_x, batch_y)
catch e:
println e.path
end
mut classes := Vector {} as Vector<str>
println data.LoadImageFolder("missing-folder", 8, 8, true, t, labels, classes)
it.Free()
set.Free()
endReset
func Reset(epoch: i64)reset
func reset(epoch: i64)The example for this item is shown with the ones above: data-Batch.salam
Next
func Next(x_out &: matrix.Matrix, y_out &: matrix.Matrix): boolnext
func next(x_out &: matrix.Matrix, y_out &: matrix.Matrix): boolx_out and y_out; the last batch may be smaller. False when no batch is left.The example for this item is shown with the ones above: data-Batch.salam
NextTensor
func NextTensor(x_out &: tensor.Tensor, labels_out &: Vector<int>): boolnext_tensor
func next_tensor(x_out &: tensor.Tensor, labels_out &: Vector<int>): boolThe example for this item is shown with the ones above: data-Batch.salam
BatchesPerEpoch
func BatchesPerEpoch(): intbatches_per_epoch
func batches_per_epoch(): intThe example for this item is shown with the ones above: data-Batch.salam
Free
func Free()free
func free()The example for this item is shown with the ones above: data-Batch.salam
MatrixDataset
struct MatrixDataset| Name | Type | Description |
|---|---|---|
x | matrix.Matrix | The feature matrix, one row per sample. |
y | matrix.Matrix | The labels, one row per sample. |
The example for this item is shown with the ones above: data-Batch.salam
Len
func Len(): intlen
func len(): intThe example for this item is shown with the ones above: data-Batch.salam
Batches
func Batches(batch: int, shuffle: bool, seed: i64): BatchIterbatches
func batches(batch: int, shuffle: bool, seed: i64): BatchItershuffle is true the order is shuffled from seed.The example for this item is shown with the ones above: data-Batch.salam
Free
func Free()free
func free()The example for this item is shown with the ones above: data-Batch.salam
Table
struct Table| Name | Type | Description |
|---|---|---|
names | Vector | The column names. |
kinds | Vector | The kind of each column; one of the COL_ constants. |
cats | Vector | The levels of each categorical column. |
raw | Vector | The original text of the cells of COL_STR columns; empty for the other kinds. |
cells | matrix.Matrix | The numeric value of the cells; a categorical column holds the level number. |
import data
import matrix
func main throws:
text := "age,city,score,ok\n30,tehran,1.5,1\n25,shiraz,2.5,0\n41,tehran,3.0,1\n"
mut t := try data.LoadCsv(text, true)
println t.rows(), t.cols(), t.names, t.ColIndex("score"), t.ColIndex("x")
println t.kinds[0] == data.COL_INT, t.kinds[1] == data.COL_CAT, t.kinds[2] == data.COL_FLOAT, data.COL_STR
println t.CatLevels("city")
mut m := t.ToMatrix()
println m.rows, m.cols, m.at(2, 1)
mut x := matrix.Matrix {}
mut y := matrix.Matrix {}
println t.SplitXy("ok", x, y), x.cols, y.at(1, 0)
try:
data.LoadCsvFile("missing.csv", true)
catch e:
println e.kind_name(), e.path
end
matrix.Free(m)
matrix.Free(x)
matrix.Free(y)
t.Free()
endRows
func Rows(): introws
func rows(): intThe example for this item is shown with the ones above: data-Table.salam
Cols
func Cols(): intcols
func cols(): intThe example for this item is shown with the ones above: data-Table.salam
ColIndex
func ColIndex(name: str): intcol_index
func col_index(name: str): intThe example for this item is shown with the ones above: data-Table.salam
CatLevels
func CatLevels(name: str): Vector<str>cat_levels
func cat_levels(name: str): Vector<str>The example for this item is shown with the ones above: data-Table.salam
ToMatrix
func ToMatrix(): matrix.Matrixto_matrix
func to_matrix(): matrix.MatrixThe example for this item is shown with the ones above: data-Table.salam
SplitXY
func SplitXY(label: str, x &: matrix.Matrix, y &: matrix.Matrix): boolSplitXy
func SplitXy(label: str, x &: matrix.Matrix, y &: matrix.Matrix): boolsplit_xy
func split_xy(label: str, x &: matrix.Matrix, y &: matrix.Matrix): booly and the rest into x.The example for this item is shown with the ones above: data-Table.salam
Free
func Free()free
func free()The example for this item is shown with the ones above: data-Table.salam
Constants and variables
Column kinds COL_…
| Name | Value |
|---|---|
COL_FLOAT | 0 |
COL_INT | 1 |
COL_CAT | 2 |
COL_STR | 3 |
The example for this item is shown with the ones above: data-Table.salam
Functions
XTensor
func XTensor(x &: matrix.Matrix): tensor.TensorThe example for this item is shown with the ones above: data-Batch.salam
YLabels
func YLabels(y &: matrix.Matrix): Vector<int>The example for this item is shown with the ones above: data-Batch.salam
LoadImageFolder
func LoadImageFolder(root: str, width: int, height: int, gray: bool, x &: tensor.Tensor, labels &: Vector<int>, classes &: Vector<str>): boolgray is true.The example for this item is shown with the ones above: data-Batch.salam
ParseIdx
func ParseIdx(p: void*, n: int): Throws<matrix.Matrix>The example for this item is shown with the ones above: data-Batch.salam
LoadIdx
func LoadIdx(path: str): Throws<matrix.Matrix>.gz) is unpacked first. Throws NotFound for a missing file and the errors of ParseIdx otherwise.The example for this item is shown with the ones above: data-Batch.salam
LoadMnist
func LoadMnist(images_path: str, labels_path: str, x &: matrix.Matrix, y &: matrix.Matrix): Throws<bool>x and y are only replaced on success.The example for this item is shown with the ones above: data-Batch.salam
LoadCsv
func LoadCsv(text: str, header: bool): Throws<Table>header false the columns are named col0, col1 and so on. Throws the encoding.csv errors, with line and column, on text it cannot read.The example for this item is shown with the ones above: data-Table.salam
LoadCsvFile
func LoadCsvFile(path: str, header: bool): Throws<Table>NotFound for a missing file and the errors of LoadCsv otherwise, with the file name in path.The example for this item is shown with the ones above: data-Table.salam
MakeBlobs
func MakeBlobs(n: int, centers: int, p: int, spread: f64, seed: i64, y &: matrix.Matrix): matrix.Matriximport data
import matrix
func count_ones(y: matrix.Matrix): int:
mut n := 0
repeat y.rows in i:
if y.at(i, 0) == 1.0:
n++
end
end
ret n
end
func main:
mut y := matrix.Matrix {}
blobs := data.MakeBlobs(90, 3, 2, 0.5, 42, y)
println blobs.rows, blobs.cols, y.rows
moons := data.MakeMoons(100, 0.1, 7, y)
println moons.rows, moons.cols, count_ones(y)
circles := data.MakeCircles(60, 0.05, 0.5, 7, y)
println circles.rows, count_ones(y)
mut coef := matrix.Matrix {}
reg := data.MakeRegression(50, 3, 0.1, 1, y, coef)
println reg.rows, reg.cols, coef.rows
cls := data.MakeClassification(40, 4, 2, 1.0, 3, y)
println cls.rows, cls.cols, count_ones(y) > 0
endMakeMoons
func MakeMoons(n: int, noise: f64, seed: i64, y &: matrix.Matrix): matrix.MatrixThe example for this item is shown with the ones above: data-Synth.salam
MakeCircles
func MakeCircles(n: int, noise: f64, factor: f64, seed: i64, y &: matrix.Matrix): matrix.Matrixfactor is the radius ratio of the inner circle.The example for this item is shown with the ones above: data-Synth.salam
MakeRegression
func MakeRegression(n: int, p: int, noise: f64, seed: i64, y &: matrix.Matrix, true_coef &: matrix.Matrix): matrix.Matrixtrue_coef.The example for this item is shown with the ones above: data-Synth.salam
MakeClassification
func MakeClassification(n: int, p: int, classes: int, sep: f64, seed: i64, y &: matrix.Matrix): matrix.Matrixsep sets how far apart the classes are.The example for this item is shown with the ones above: data-Synth.salam