Standard library · Encoding and data

encoding.binary

Writing and reading numbers as bytes, big-endian or little-endian.

import encoding.binary

Binary file formats and network protocols store each number as a fixed number of bytes in a fixed order. This package writes and reads such fields: 8, 16, 32 and 64 bit integers, signed or unsigned, and 32 or 64 bit IEEE 754 floats. Big-endian (BE) puts the most significant byte first and is the usual order on the network and in PNG; little-endian (LE) is the other way round and is what x86 and ARM processors use, and formats such as ZIP and WAV.

Writers append to a ByteBuf and readers advance a ByteReader, both from compress; that package also has BufPushBytes and RdBytes for raw byte runs such as magic numbers. Use this package when the layout is fixed by a specification or by you. For self-describing data use encoding.msgpack or encoding.protobuf; for hex or Base64 text, encoding.

For random access into bytes you already have, such as a file read whole or a memory-mapped binary, use a View: GetU32LE(view, offset) and its relatives read a number at any offset, PutU32LE and its relatives write one, and CStringAt and FixedStringAt read the names that executable formats store. A view checks every access against its length: a getter returns 0 outside it and a setter returns false, so a damaged header cannot read beyond the buffer; call Has first when 0 is a legal value.

What trips people up

  • Every Read... function is throws. When fewer bytes are left than the value needs, it throws a core Error of kind UnexpectedEnd with the code binary.unexpected_end and the reader's position in offset, and the reader does not move. Call it with try inside a decoder that is itself throws, so a truncated file stops with an error instead of decoding zeros. The check costs one comparison per value; a loop of 10 million ReadU32LE calls took about 3 ms with the LLVM backend and about 100 ms with the C backend (gcc or tcc) on a busy 6 core x86-64 machine.
  • 32-bit unsigned values come back in an int, which is signed, so values from 2,147,483,648 up look negative. ReadU32BE and ReadI32BE return the same bits. Widen with (v as i64) & 4294967295.
  • 64-bit unsigned is an i64 too, so values above 9,223,372,036,854,775,807 are negative.
  • Writers do not range check. WriteU8(b, 300) writes 44, the low 8 bits.
  • BinaryOptions is not used by any function yet; the float functions take a big_endian bool instead, and the integer functions carry the order in their names.

Guide: your own file format

A binary format usually starts with a few magic bytes that identify it, then a version number and a count, then the records. Write each field with the function for its size and byte order, and read them back in exactly the same order. compress.BufPushBytes writes a fixed text such as the magic, and compress.RdBytes reads it back.

encoding-binary-guide-header.salam
import encoding.binary
import compress
import str

func write_file(scores: Vector<int>): compress.ByteBuf:
    mut b := compress.NewByteBuf()
    compress.BufPushBytes(b, "SLM1" as void*, 4)
    binary.WriteU16LE(b, 2)
    binary.WriteU32LE(b, scores.Len())
    each s in scores:
        binary.WriteI32LE(b, s)
    end
    ret b
end

func read_file(data: compress.ByteBuf, size: int): str throws:
    mut r := compress.NewByteReader(data.data, size)
    mut head := compress.RdBytes(r, 4)
    defer compress.BufFree(head)
    if compress.BufToStrUnsafe(head) != "SLM1":
        throw Error.At(ErrorKind.Syntax, "not our file", 0)
    end
    version := try binary.ReadU16LE(r)
    if version != 2:
        throw Error.At(ErrorKind.Unsupported, "unknown version " + str.FromInt(version), 4)
    end
    count := try binary.ReadU32LE(r)
    if count < 0 or count > 100000:
        throw Error.At(ErrorKind.OutOfRange, "implausible count", 6)
    end
    mut total := 0 as i64
    repeat count:
        total += (try binary.ReadI32LE(r)) as i64
    end
    ret str.FromInt(count) + " scores, total " + str.FromInt(total)
end

func show(label: str, data: compress.ByteBuf, size: int):
    try:
        println label, read_file(data, size)
    catch e:
        println label, e.ToStr()
    end
end

func main:
    mut scores := Vector {} as Vector<int>
    scores.Push(120)
    scores.Push(-15)
    scores.Push(98000)
    mut data := write_file(scores)
    println "file is", compress.BufLen(data), "bytes"

    show("whole file:", data, compress.BufLen(data))
    show("first 18:  ", data, 18)
    compress.BufSet(data, 4, 9)
    show("edited:    ", data, compress.BufLen(data))
    compress.BufFree(data)
end
Outputfile is 22 bytes whole file: 3 scores, total 98105 first 18: unexpected end of data: needed 4 bytes, 0 left (offset 18) edited: unknown version 9 (offset 4)

read_file checks the magic and the version before trusting the rest, and refuses a count that is too large to be real, so a damaged file cannot make it loop for a long time. It then reads every record with try, so the 18-byte copy, which has room for only two of the three scores, stops at the third with the offset where the data ran out. The checks of its own throw an Error too, so the caller handles every failure in one catch. Changing byte 4 turns the version into 9, which is refused. Keep one byte order for the whole format and write it down in its description.

Guide: read a header of a real format

PNG is a good example of a big-endian format. A file starts with an 8 byte signature, then chunks, each with a 4 byte length, a 4 letter type and the data. The first chunk, IHDR, holds the width and height as big-endian 32 bit numbers. The program builds those first bytes itself so it runs without a file; with a real image, read the file into a buffer first and the parsing is the same.

encoding-binary-guide-png.salam
import encoding.binary
import compress

func fake_png(): compress.ByteBuf:
    mut b := compress.NewByteBuf()
    sig := [137, 80, 78, 71, 13, 10, 26, 10]
    each x in sig:
        binary.WriteU8(b, x)
    end
    binary.WriteU32BE(b, 13)
    compress.BufPushBytes(b, "IHDR" as void*, 4)
    binary.WriteU32BE(b, 1280)
    binary.WriteU32BE(b, 720)
    binary.WriteU8(b, 8)
    binary.WriteU8(b, 6)
    ret b
end

func main throws:
    mut png := fake_png()
    defer compress.BufFree(png)
    mut r := compress.ReaderFromBuf(png)
    b0 := try binary.ReadU8(r)
    b1 := try binary.ReadU8(r)
    if b0 != 137 or b1 != 80:
        throw "not a PNG"
    end
    repeat 6:
        _ := try binary.ReadU8(r)
    end
    length := try binary.ReadU32BE(r)
    mut kind := compress.RdBytes(r, 4)
    println "first chunk:", compress.BufToStrUnsafe(kind), "of", length, "bytes"
    compress.BufFree(kind)
    width := try binary.ReadU32BE(r)
    height := try binary.ReadU32BE(r)
    println "size:", width, "x", height
    depth := try binary.ReadU8(r)
    color := try binary.ReadU8(r)
    println "bit depth:", depth, "color type:", color
    try:
        binary.ReadU8(r)
    catch e:
        println "the rest of IHDR is missing:", e.ToStr()
    end
end
Outputfirst chunk: IHDR of 13 bytes size: 1280 x 720 bit depth: 8 color type: 6 the rest of IHDR is missing: unexpected end of data: needed 1 byte, 0 left (offset 26)

Guide: length-prefixed messages

A TCP connection delivers a stream of bytes, not separate messages. The usual fix is to send each message with its length in front, here as a 16 bit big-endian number. The reader takes the length, checks that the whole message has arrived with RdLeft, and only then reads it.

encoding-binary-guide-frames.salam
import encoding.binary
import compress

func send(b &: compress.ByteBuf, msg: str):
    binary.WriteU16BE(b, msg.Len())
    compress.BufPushBytes(b, msg as void*, msg.Len())
end

func main throws:
    mut wire := compress.NewByteBuf()
    send(wire, "HELLO")
    send(wire, "سلام")
    send(wire, "")
    send(wire, "BYE")
    binary.WriteU16BE(wire, 10)
    compress.BufPushBytes(wire, "par" as void*, 3)
    println "on the wire:", compress.BufLen(wire), "bytes"

    mut r := compress.ReaderFromBuf(wire)
    until compress.RdLeft(r) >= 2:
        n := try binary.ReadU16BE(r)
        if compress.RdLeft(r) < n:
            println "incomplete frame, wait for more data"
            break
        end
        mut body := compress.RdBytes(r, n)
        println n, "bytes: [" + compress.BufToStrUnsafe(body) + "]"
        compress.BufFree(body)
    end
    compress.BufFree(wire)
end
Outputon the wire: 29 bytes 5 bytes: [HELLO] 8 bytes: [سلام] 0 bytes: [] 3 bytes: [BYE] incomplete frame, wait for more data

The length counts bytes, so سلام is 8, not 4. The last frame says 10 bytes but only 3 have arrived; a real server keeps those bytes and tries again when more data comes in.

Guide: byte order and sign mistakes

Reading with the wrong byte order does not fail, it just gives a wrong number, so these bugs are easy to miss. Unsigned 32 bit values are another trap: they come back in an int, so anything at or above 2,147,483,648 looks negative. Widen it to i64 and mask it to get the unsigned value.

encoding-binary-guide-endian.salam
import encoding.binary
import compress
import str

func bytes_of(b: compress.ByteBuf): str:
    mut out := ""
    repeat compress.BufLen(b) in i:
        if i > 0:
            out += " "
        end
        out += str.FromInt(compress.BufGet(b, i))
    end
    ret out
end

func main throws:
    mut be := compress.NewByteBuf()
    binary.WriteU32BE(be, 1000)
    mut le := compress.NewByteBuf()
    binary.WriteU32LE(le, 1000)
    println "1000 big-endian:   ", bytes_of(be)
    println "1000 little-endian:", bytes_of(le)

    mut r := compress.ReaderFromBuf(le)
    println "read with the wrong order:", try binary.ReadU32BE(r)

    mut big := compress.NewByteBuf()
    binary.WriteU32BE(big, -1)
    mut r2 := compress.ReaderFromBuf(big)
    v := try binary.ReadU32BE(r2)
    println "u32 0xFFFFFFFF as int:", v, "as unsigned:", (v as i64) & 4294967295

    mut r3 := compress.ReaderFromBuf(be)
    _ := try binary.ReadU32BE(r3)
    println "bytes left after reading:", compress.RdLeft(r3)
    try:
        binary.ReadU16BE(r3)
    catch e:
        println "one more read:", e.ToStr(), e.code
    end

    compress.BufFree(big)
    compress.BufFree(le)
    compress.BufFree(be)
end
Output1000 big-endian: 0 0 3 232 1000 little-endian: 232 3 0 0 read with the wrong order: -402456576 u32 0xFFFFFFFF as int: -1 as unsigned: 4294967295 bytes left after reading: 0 one more read: unexpected end of data: needed 2 bytes, 0 left (offset 4) binary.unexpected_end

Reading past the end, on the other hand, is never silent: the last read throws binary.unexpected_end. Check compress.RdLeft first when running out of data is normal rather than an error, as the framing example does for a frame that has not fully arrived.

Types

View

struct View
A pointer and a length: a window on bytes that something else owns, such as a ByteBuf or memory from mem. The getters and setters below check every access against the length.
NameTypeDescription
datavoid*The first byte of the window.
lenint = 0The number of bytes in the window.
encoding-binary-View.salam
import encoding.binary
import mem

func main:
    raw := mem.AllocateZeroed(32 as u64)
    defer mem.Free(raw)
    v := binary.NewView(raw, 32)
    binary.PutU32LE(v, 0, 0xFEEDFACF as i64)
    binary.PutU16BE(v, 4, 0x0102 as i64)
    binary.PutU8(v, 8, 65 as i64)
    binary.PutU8(v, 9, 66 as i64)
    println binary.GetU32LE(v, 0), binary.GetU16BE(v, 4), binary.GetU16LE(v, 4)
    println binary.CStringAt(v, 8), binary.FixedStringAt(v, 8, 1)
    println binary.Has(v, 30, 2), binary.Has(v, 31, 2), binary.GetU32LE(v, 31)
end
Output4277009103 258 513 AB A true false 0

BinaryOptions

struct BinaryOptions
Byte order options. No function in the package takes this struct yet.
NameTypeDescription
big_endianbool = falseBig-endian when true; false by default.
encoding-binary-ReadWrite.salam
import encoding.binary
import compress

func main throws:
    mut b := compress.NewByteBuf()
    binary.WriteU8(b, 255)
    binary.WriteU16BE(b, 4660)
    binary.WriteU16LE(b, 4660)
    binary.WriteU32BE(b, 1)
    binary.WriteI16LE(b, -2)
    println compress.BufToBase64(b)
    binary.WriteU32LE(b, 7)
    binary.WriteU64BE(b, 8)
    binary.WriteU64LE(b, 9)
    binary.WriteI16BE(b, -3)
    binary.WriteI32BE(b, -4)
    binary.WriteI32LE(b, -5)
    binary.WriteI64BE(b, -6)
    binary.WriteI64LE(b, -7)
    binary.WriteF64(b, 1.5, true)
    binary.WriteF32(b, 0.25, false)
    println compress.BufLen(b)
    mut r := compress.ReaderFromBuf(b)
    println try binary.ReadU8(r), try binary.ReadU16BE(r), try binary.ReadU16LE(r), try binary.ReadU32BE(r), try binary.ReadI16LE(r)
    println try binary.ReadU32LE(r), try binary.ReadU64BE(r), try binary.ReadU64LE(r)
    println try binary.ReadI16BE(r), try binary.ReadI32BE(r), try binary.ReadI32LE(r), try binary.ReadI64BE(r), try binary.ReadI64LE(r)
    println try binary.ReadF64(r, true), try binary.ReadF32(r, false)
    try:
        binary.ReadU8(r)
    catch e:
        println e.kind_name(), e.offset, e.code
    end
    o := binary.BinaryOptions {}
    println o.big_endian
    compress.BufFree(b)
end
Output/xI0NBIAAAAB/v8= 69 255 4660 4660 1 -2 7 8 9 -3 -4 -5 -6 -7 1.5 0.25 unexpected end of input 69 binary.unexpected_end false

Functions

NewView

func NewView(p: void*, n: int): View
Makes a view of n bytes starting at p.

The example for this item is shown with the ones above: encoding-binary-View.salam

ViewOf

func ViewOf(b: compress.ByteBuf): View
Makes a view of the bytes in a ByteBuf. It does not copy; keep the buffer alive and unchanged in size while you use the view.

The example for this item is shown with the ones above: encoding-binary-View.salam

Has

func Has(v: View, off: int, n: int): bool
True when the n bytes from offset off lie inside the view.

The example for this item is shown with the ones above: encoding-binary-View.salam

GetU8

func GetU8(v: View, off: int): int
The byte at off; 0 when it is outside the view. Use Has when 0 is a possible value.

The example for this item is shown with the ones above: encoding-binary-View.salam

GetU16LE

func GetU16LE(v: View, off: int): int
The unsigned 16 bit integer at off, little-endian; 0 outside the view.

The example for this item is shown with the ones above: encoding-binary-View.salam

GetU16BE

func GetU16BE(v: View, off: int): int
The unsigned 16 bit integer at off, big-endian; 0 outside the view.

The example for this item is shown with the ones above: encoding-binary-View.salam

GetU32LE

func GetU32LE(v: View, off: int): i64
The unsigned 32 bit integer at off, little-endian, as an i64 so values above 2^31 stay positive; 0 outside the view.

The example for this item is shown with the ones above: encoding-binary-View.salam

GetU32BE

func GetU32BE(v: View, off: int): i64
The unsigned 32 bit integer at off, big-endian; 0 outside the view.

The example for this item is shown with the ones above: encoding-binary-View.salam

GetU64LE

func GetU64LE(v: View, off: int): i64
The 64 bit integer at off, little-endian (values above 2^63 appear negative); 0 outside the view.

The example for this item is shown with the ones above: encoding-binary-View.salam

GetU64BE

func GetU64BE(v: View, off: int): i64
The 64 bit integer at off, big-endian; 0 outside the view.

The example for this item is shown with the ones above: encoding-binary-View.salam

PutU8

func PutU8(v: View, off: int, x: i64): bool
Writes the low 8 bits of x at off; false, with nothing written, when it does not fit in the view.

The example for this item is shown with the ones above: encoding-binary-View.salam

PutU16LE

func PutU16LE(v: View, off: int, x: i64): bool
Writes the low 16 bits of x little-endian at off; false when it does not fit.

The example for this item is shown with the ones above: encoding-binary-View.salam

PutU16BE

func PutU16BE(v: View, off: int, x: i64): bool
Writes the low 16 bits of x big-endian at off; false when it does not fit.

The example for this item is shown with the ones above: encoding-binary-View.salam

PutU32LE

func PutU32LE(v: View, off: int, x: i64): bool
Writes the low 32 bits of x little-endian at off; false when it does not fit.

The example for this item is shown with the ones above: encoding-binary-View.salam

PutU32BE

func PutU32BE(v: View, off: int, x: i64): bool
Writes the low 32 bits of x big-endian at off; false when it does not fit.

The example for this item is shown with the ones above: encoding-binary-View.salam

PutU64LE

func PutU64LE(v: View, off: int, x: i64): bool
Writes x as 64 bits little-endian at off; false when it does not fit.

The example for this item is shown with the ones above: encoding-binary-View.salam

PutU64BE

func PutU64BE(v: View, off: int, x: i64): bool
Writes x as 64 bits big-endian at off; false when it does not fit.

The example for this item is shown with the ones above: encoding-binary-View.salam

CStringAt

func CStringAt(v: View, off: int): str
The text that starts at off and ends at the first zero byte or the end of the view, as a new string. Empty outside the view.

The example for this item is shown with the ones above: encoding-binary-View.salam

FixedStringAt

func FixedStringAt(v: View, off: int, width: int): str
The text in a field of exactly width bytes at off, cut at the first zero byte (a Mach-O segment name or an ELF section name, for example). Empty when the field does not fit.

The example for this item is shown with the ones above: encoding-binary-View.salam

WriteU8

func WriteU8(b &: compress.ByteBuf, v: int)
Writes one byte: the low 8 bits of v.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU16BE

func WriteU16BE(b &: compress.ByteBuf, v: int)
Writes an unsigned 16 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU16LE

func WriteU16LE(b &: compress.ByteBuf, v: int)
Writes an unsigned 16 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU32BE

func WriteU32BE(b &: compress.ByteBuf, v: int)
Writes an unsigned 32 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU32LE

func WriteU32LE(b &: compress.ByteBuf, v: int)
Writes an unsigned 32 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU64BE

func WriteU64BE(b &: compress.ByteBuf, v: i64)
Writes an unsigned 64 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteU64LE

func WriteU64LE(b &: compress.ByteBuf, v: i64)
Writes an unsigned 64 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI16BE

func WriteI16BE(b &: compress.ByteBuf, v: int)
Writes a signed 16 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI16LE

func WriteI16LE(b &: compress.ByteBuf, v: int)
Writes a signed 16 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI32BE

func WriteI32BE(b &: compress.ByteBuf, v: int)
Writes a signed 32 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI32LE

func WriteI32LE(b &: compress.ByteBuf, v: int)
Writes a signed 32 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI64BE

func WriteI64BE(b &: compress.ByteBuf, v: i64)
Writes a signed 64 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteI64LE

func WriteI64LE(b &: compress.ByteBuf, v: i64)
Writes a signed 64 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteF64

func WriteF64(b &: compress.ByteBuf, v: f64, big_endian: bool)
Writes a 64 bit IEEE 754 float; big_endian picks the byte order.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

WriteF32

func WriteF32(b &: compress.ByteBuf, v: f64, big_endian: bool)
Writes a 32 bit float; the value loses precision down to 32 bits as it is written.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU8

func ReadU8(r &: compress.ByteReader): Throws<int>
Reads one byte, 0 to 255. Like every reader here it throws UnexpectedEnd (binary.unexpected_end, with the offset) when the data has ended, and leaves the reader where it was.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU16BE

func ReadU16BE(r &: compress.ByteReader): Throws<int>
Reads an unsigned 16 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU16LE

func ReadU16LE(r &: compress.ByteReader): Throws<int>
Reads an unsigned 16 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU32BE

func ReadU32BE(r &: compress.ByteReader): Throws<int>
Reads a 32 bit integer big-endian, most significant byte first. The result is an int, so values from 231 up come back negative; mask with (v as i64) & 4294967295.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU32LE

func ReadU32LE(r &: compress.ByteReader): Throws<int>
Reads a 32 bit integer little-endian, least significant byte first. Values from 231 up come back negative, as with ReadU32BE.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU64BE

func ReadU64BE(r &: compress.ByteReader): Throws<i64>
Reads an unsigned 64 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadU64LE

func ReadU64LE(r &: compress.ByteReader): Throws<i64>
Reads an unsigned 64 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI16BE

func ReadI16BE(r &: compress.ByteReader): Throws<int>
Reads a signed 16 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI16LE

func ReadI16LE(r &: compress.ByteReader): Throws<int>
Reads a signed 16 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI32BE

func ReadI32BE(r &: compress.ByteReader): Throws<int>
Reads a signed 32 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI32LE

func ReadI32LE(r &: compress.ByteReader): Throws<int>
Reads a signed 32 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI64BE

func ReadI64BE(r &: compress.ByteReader): Throws<i64>
Reads a signed 64 bit integer big-endian, most significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadI64LE

func ReadI64LE(r &: compress.ByteReader): Throws<i64>
Reads a signed 64 bit integer little-endian, least significant byte first.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadF64

func ReadF64(r &: compress.ByteReader, big_endian: bool): Throws<f64>
Reads a 64 bit IEEE 754 float; big_endian picks the byte order.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam

ReadF32

func ReadF32(r &: compress.ByteReader, big_endian: bool): Throws<f64>
Reads a 32 bit float and returns it as a 64 bit float; big_endian picks the byte order.

The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam