encoding.binary
Writing and reading numbers as bytes, big-endian or little-endian.
import encoding.binary
Binary file formats and network protocols store each number as a fixed number of bytes in a fixed order. This package writes and reads such fields: 8, 16, 32 and 64 bit integers, signed or unsigned, and 32 or 64 bit IEEE 754 floats. Big-endian (BE) puts the most significant byte first and is the usual order on the network and in PNG; little-endian (LE) is the other way round and is what x86 and ARM processors use, and formats such as ZIP and WAV.
Writers append to a ByteBuf and readers advance a ByteReader, both from compress; that package also has BufPushBytes and RdBytes for raw byte runs such as magic numbers. Use this package when the layout is fixed by a specification or by you. For self-describing data use encoding.msgpack or encoding.protobuf; for hex or Base64 text, encoding.
For random access into bytes you already have, such as a file read whole or a memory-mapped binary, use a View: GetU32LE(view, offset) and its relatives read a number at any offset, PutU32LE and its relatives write one, and CStringAt and FixedStringAt read the names that executable formats store. A view checks every access against its length: a getter returns 0 outside it and a setter returns false, so a damaged header cannot read beyond the buffer; call Has first when 0 is a legal value.
What trips people up
- Every
Read...function isthrows. When fewer bytes are left than the value needs, it throws a coreErrorof kindUnexpectedEndwith the codebinary.unexpected_endand the reader's position inoffset, and the reader does not move. Call it withtryinside a decoder that is itselfthrows, so a truncated file stops with an error instead of decoding zeros. The check costs one comparison per value; a loop of 10 millionReadU32LEcalls took about 3 ms with the LLVM backend and about 100 ms with the C backend (gcc or tcc) on a busy 6 core x86-64 machine. - 32-bit unsigned values come back in an
int, which is signed, so values from 2,147,483,648 up look negative.ReadU32BEandReadI32BEreturn the same bits. Widen with(v as i64) & 4294967295. - 64-bit unsigned is an
i64too, so values above 9,223,372,036,854,775,807 are negative. - Writers do not range check.
WriteU8(b, 300)writes 44, the low 8 bits. BinaryOptionsis not used by any function yet; the float functions take abig_endianbool instead, and the integer functions carry the order in their names.
Guide: your own file format
A binary format usually starts with a few magic bytes that identify it, then a version number and a count, then the records. Write each field with the function for its size and byte order, and read them back in exactly the same order. compress.BufPushBytes writes a fixed text such as the magic, and compress.RdBytes reads it back.
import encoding.binary
import compress
import str
func write_file(scores: Vector<int>): compress.ByteBuf:
mut b := compress.NewByteBuf()
compress.BufPushBytes(b, "SLM1" as void*, 4)
binary.WriteU16LE(b, 2)
binary.WriteU32LE(b, scores.Len())
each s in scores:
binary.WriteI32LE(b, s)
end
ret b
end
func read_file(data: compress.ByteBuf, size: int): str throws:
mut r := compress.NewByteReader(data.data, size)
mut head := compress.RdBytes(r, 4)
defer compress.BufFree(head)
if compress.BufToStrUnsafe(head) != "SLM1":
throw Error.At(ErrorKind.Syntax, "not our file", 0)
end
version := try binary.ReadU16LE(r)
if version != 2:
throw Error.At(ErrorKind.Unsupported, "unknown version " + str.FromInt(version), 4)
end
count := try binary.ReadU32LE(r)
if count < 0 or count > 100000:
throw Error.At(ErrorKind.OutOfRange, "implausible count", 6)
end
mut total := 0 as i64
repeat count:
total += (try binary.ReadI32LE(r)) as i64
end
ret str.FromInt(count) + " scores, total " + str.FromInt(total)
end
func show(label: str, data: compress.ByteBuf, size: int):
try:
println label, read_file(data, size)
catch e:
println label, e.ToStr()
end
end
func main:
mut scores := Vector {} as Vector<int>
scores.Push(120)
scores.Push(-15)
scores.Push(98000)
mut data := write_file(scores)
println "file is", compress.BufLen(data), "bytes"
show("whole file:", data, compress.BufLen(data))
show("first 18: ", data, 18)
compress.BufSet(data, 4, 9)
show("edited: ", data, compress.BufLen(data))
compress.BufFree(data)
endread_file checks the magic and the version before trusting the rest, and refuses a count that is too large to be real, so a damaged file cannot make it loop for a long time. It then reads every record with try, so the 18-byte copy, which has room for only two of the three scores, stops at the third with the offset where the data ran out. The checks of its own throw an Error too, so the caller handles every failure in one catch. Changing byte 4 turns the version into 9, which is refused. Keep one byte order for the whole format and write it down in its description.
Guide: read a header of a real format
PNG is a good example of a big-endian format. A file starts with an 8 byte signature, then chunks, each with a 4 byte length, a 4 letter type and the data. The first chunk, IHDR, holds the width and height as big-endian 32 bit numbers. The program builds those first bytes itself so it runs without a file; with a real image, read the file into a buffer first and the parsing is the same.
import encoding.binary
import compress
func fake_png(): compress.ByteBuf:
mut b := compress.NewByteBuf()
sig := [137, 80, 78, 71, 13, 10, 26, 10]
each x in sig:
binary.WriteU8(b, x)
end
binary.WriteU32BE(b, 13)
compress.BufPushBytes(b, "IHDR" as void*, 4)
binary.WriteU32BE(b, 1280)
binary.WriteU32BE(b, 720)
binary.WriteU8(b, 8)
binary.WriteU8(b, 6)
ret b
end
func main throws:
mut png := fake_png()
defer compress.BufFree(png)
mut r := compress.ReaderFromBuf(png)
b0 := try binary.ReadU8(r)
b1 := try binary.ReadU8(r)
if b0 != 137 or b1 != 80:
throw "not a PNG"
end
repeat 6:
_ := try binary.ReadU8(r)
end
length := try binary.ReadU32BE(r)
mut kind := compress.RdBytes(r, 4)
println "first chunk:", compress.BufToStrUnsafe(kind), "of", length, "bytes"
compress.BufFree(kind)
width := try binary.ReadU32BE(r)
height := try binary.ReadU32BE(r)
println "size:", width, "x", height
depth := try binary.ReadU8(r)
color := try binary.ReadU8(r)
println "bit depth:", depth, "color type:", color
try:
binary.ReadU8(r)
catch e:
println "the rest of IHDR is missing:", e.ToStr()
end
endGuide: length-prefixed messages
A TCP connection delivers a stream of bytes, not separate messages. The usual fix is to send each message with its length in front, here as a 16 bit big-endian number. The reader takes the length, checks that the whole message has arrived with RdLeft, and only then reads it.
import encoding.binary
import compress
func send(b &: compress.ByteBuf, msg: str):
binary.WriteU16BE(b, msg.Len())
compress.BufPushBytes(b, msg as void*, msg.Len())
end
func main throws:
mut wire := compress.NewByteBuf()
send(wire, "HELLO")
send(wire, "سلام")
send(wire, "")
send(wire, "BYE")
binary.WriteU16BE(wire, 10)
compress.BufPushBytes(wire, "par" as void*, 3)
println "on the wire:", compress.BufLen(wire), "bytes"
mut r := compress.ReaderFromBuf(wire)
until compress.RdLeft(r) >= 2:
n := try binary.ReadU16BE(r)
if compress.RdLeft(r) < n:
println "incomplete frame, wait for more data"
break
end
mut body := compress.RdBytes(r, n)
println n, "bytes: [" + compress.BufToStrUnsafe(body) + "]"
compress.BufFree(body)
end
compress.BufFree(wire)
endThe length counts bytes, so سلام is 8, not 4. The last frame says 10 bytes but only 3 have arrived; a real server keeps those bytes and tries again when more data comes in.
Guide: byte order and sign mistakes
Reading with the wrong byte order does not fail, it just gives a wrong number, so these bugs are easy to miss. Unsigned 32 bit values are another trap: they come back in an int, so anything at or above 2,147,483,648 looks negative. Widen it to i64 and mask it to get the unsigned value.
import encoding.binary
import compress
import str
func bytes_of(b: compress.ByteBuf): str:
mut out := ""
repeat compress.BufLen(b) in i:
if i > 0:
out += " "
end
out += str.FromInt(compress.BufGet(b, i))
end
ret out
end
func main throws:
mut be := compress.NewByteBuf()
binary.WriteU32BE(be, 1000)
mut le := compress.NewByteBuf()
binary.WriteU32LE(le, 1000)
println "1000 big-endian: ", bytes_of(be)
println "1000 little-endian:", bytes_of(le)
mut r := compress.ReaderFromBuf(le)
println "read with the wrong order:", try binary.ReadU32BE(r)
mut big := compress.NewByteBuf()
binary.WriteU32BE(big, -1)
mut r2 := compress.ReaderFromBuf(big)
v := try binary.ReadU32BE(r2)
println "u32 0xFFFFFFFF as int:", v, "as unsigned:", (v as i64) & 4294967295
mut r3 := compress.ReaderFromBuf(be)
_ := try binary.ReadU32BE(r3)
println "bytes left after reading:", compress.RdLeft(r3)
try:
binary.ReadU16BE(r3)
catch e:
println "one more read:", e.ToStr(), e.code
end
compress.BufFree(big)
compress.BufFree(le)
compress.BufFree(be)
endReading past the end, on the other hand, is never silent: the last read throws binary.unexpected_end. Check compress.RdLeft first when running out of data is normal rather than an error, as the framing example does for a frame that has not fully arrived.
Types
View
struct ViewByteBuf or memory from mem. The getters and setters below check every access against the length.| Name | Type | Description |
|---|---|---|
data | void* | The first byte of the window. |
len | int = 0 | The number of bytes in the window. |
import encoding.binary
import mem
func main:
raw := mem.AllocateZeroed(32 as u64)
defer mem.Free(raw)
v := binary.NewView(raw, 32)
binary.PutU32LE(v, 0, 0xFEEDFACF as i64)
binary.PutU16BE(v, 4, 0x0102 as i64)
binary.PutU8(v, 8, 65 as i64)
binary.PutU8(v, 9, 66 as i64)
println binary.GetU32LE(v, 0), binary.GetU16BE(v, 4), binary.GetU16LE(v, 4)
println binary.CStringAt(v, 8), binary.FixedStringAt(v, 8, 1)
println binary.Has(v, 30, 2), binary.Has(v, 31, 2), binary.GetU32LE(v, 31)
endBinaryOptions
struct BinaryOptions| Name | Type | Description |
|---|---|---|
big_endian | bool = false | Big-endian when true; false by default. |
import encoding.binary
import compress
func main throws:
mut b := compress.NewByteBuf()
binary.WriteU8(b, 255)
binary.WriteU16BE(b, 4660)
binary.WriteU16LE(b, 4660)
binary.WriteU32BE(b, 1)
binary.WriteI16LE(b, -2)
println compress.BufToBase64(b)
binary.WriteU32LE(b, 7)
binary.WriteU64BE(b, 8)
binary.WriteU64LE(b, 9)
binary.WriteI16BE(b, -3)
binary.WriteI32BE(b, -4)
binary.WriteI32LE(b, -5)
binary.WriteI64BE(b, -6)
binary.WriteI64LE(b, -7)
binary.WriteF64(b, 1.5, true)
binary.WriteF32(b, 0.25, false)
println compress.BufLen(b)
mut r := compress.ReaderFromBuf(b)
println try binary.ReadU8(r), try binary.ReadU16BE(r), try binary.ReadU16LE(r), try binary.ReadU32BE(r), try binary.ReadI16LE(r)
println try binary.ReadU32LE(r), try binary.ReadU64BE(r), try binary.ReadU64LE(r)
println try binary.ReadI16BE(r), try binary.ReadI32BE(r), try binary.ReadI32LE(r), try binary.ReadI64BE(r), try binary.ReadI64LE(r)
println try binary.ReadF64(r, true), try binary.ReadF32(r, false)
try:
binary.ReadU8(r)
catch e:
println e.kind_name(), e.offset, e.code
end
o := binary.BinaryOptions {}
println o.big_endian
compress.BufFree(b)
endFunctions
NewView
func NewView(p: void*, n: int): Viewn bytes starting at p.The example for this item is shown with the ones above: encoding-binary-View.salam
ViewOf
func ViewOf(b: compress.ByteBuf): ViewByteBuf. It does not copy; keep the buffer alive and unchanged in size while you use the view.The example for this item is shown with the ones above: encoding-binary-View.salam
Has
func Has(v: View, off: int, n: int): booln bytes from offset off lie inside the view.The example for this item is shown with the ones above: encoding-binary-View.salam
GetU8
func GetU8(v: View, off: int): intoff; 0 when it is outside the view. Use Has when 0 is a possible value.The example for this item is shown with the ones above: encoding-binary-View.salam
GetU16LE
func GetU16LE(v: View, off: int): intoff, little-endian; 0 outside the view.The example for this item is shown with the ones above: encoding-binary-View.salam
GetU16BE
func GetU16BE(v: View, off: int): intoff, big-endian; 0 outside the view.The example for this item is shown with the ones above: encoding-binary-View.salam
GetU32LE
func GetU32LE(v: View, off: int): i64off, little-endian, as an i64 so values above 2^31 stay positive; 0 outside the view.The example for this item is shown with the ones above: encoding-binary-View.salam
GetU32BE
func GetU32BE(v: View, off: int): i64off, big-endian; 0 outside the view.The example for this item is shown with the ones above: encoding-binary-View.salam
GetU64LE
func GetU64LE(v: View, off: int): i64off, little-endian (values above 2^63 appear negative); 0 outside the view.The example for this item is shown with the ones above: encoding-binary-View.salam
GetU64BE
func GetU64BE(v: View, off: int): i64off, big-endian; 0 outside the view.The example for this item is shown with the ones above: encoding-binary-View.salam
PutU8
func PutU8(v: View, off: int, x: i64): boolx at off; false, with nothing written, when it does not fit in the view.The example for this item is shown with the ones above: encoding-binary-View.salam
PutU16LE
func PutU16LE(v: View, off: int, x: i64): boolx little-endian at off; false when it does not fit.The example for this item is shown with the ones above: encoding-binary-View.salam
PutU16BE
func PutU16BE(v: View, off: int, x: i64): boolx big-endian at off; false when it does not fit.The example for this item is shown with the ones above: encoding-binary-View.salam
PutU32LE
func PutU32LE(v: View, off: int, x: i64): boolx little-endian at off; false when it does not fit.The example for this item is shown with the ones above: encoding-binary-View.salam
PutU32BE
func PutU32BE(v: View, off: int, x: i64): boolx big-endian at off; false when it does not fit.The example for this item is shown with the ones above: encoding-binary-View.salam
PutU64LE
func PutU64LE(v: View, off: int, x: i64): boolx as 64 bits little-endian at off; false when it does not fit.The example for this item is shown with the ones above: encoding-binary-View.salam
PutU64BE
func PutU64BE(v: View, off: int, x: i64): boolx as 64 bits big-endian at off; false when it does not fit.The example for this item is shown with the ones above: encoding-binary-View.salam
CStringAt
func CStringAt(v: View, off: int): stroff and ends at the first zero byte or the end of the view, as a new string. Empty outside the view.The example for this item is shown with the ones above: encoding-binary-View.salam
FixedStringAt
func FixedStringAt(v: View, off: int, width: int): strwidth bytes at off, cut at the first zero byte (a Mach-O segment name or an ELF section name, for example). Empty when the field does not fit.The example for this item is shown with the ones above: encoding-binary-View.salam
WriteU8
func WriteU8(b &: compress.ByteBuf, v: int)v.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU16BE
func WriteU16BE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU16LE
func WriteU16LE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU32BE
func WriteU32BE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU32LE
func WriteU32LE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU64BE
func WriteU64BE(b &: compress.ByteBuf, v: i64)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteU64LE
func WriteU64LE(b &: compress.ByteBuf, v: i64)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI16BE
func WriteI16BE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI16LE
func WriteI16LE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI32BE
func WriteI32BE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI32LE
func WriteI32LE(b &: compress.ByteBuf, v: int)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI64BE
func WriteI64BE(b &: compress.ByteBuf, v: i64)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteI64LE
func WriteI64LE(b &: compress.ByteBuf, v: i64)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteF64
func WriteF64(b &: compress.ByteBuf, v: f64, big_endian: bool)big_endian picks the byte order.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
WriteF32
func WriteF32(b &: compress.ByteBuf, v: f64, big_endian: bool)The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU8
func ReadU8(r &: compress.ByteReader): Throws<int>UnexpectedEnd (binary.unexpected_end, with the offset) when the data has ended, and leaves the reader where it was.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU16BE
func ReadU16BE(r &: compress.ByteReader): Throws<int>The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU16LE
func ReadU16LE(r &: compress.ByteReader): Throws<int>The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU32BE
func ReadU32BE(r &: compress.ByteReader): Throws<int>int, so values from 231 up come back negative; mask with (v as i64) & 4294967295.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU32LE
func ReadU32LE(r &: compress.ByteReader): Throws<int>ReadU32BE.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU64BE
func ReadU64BE(r &: compress.ByteReader): Throws<i64>The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadU64LE
func ReadU64LE(r &: compress.ByteReader): Throws<i64>The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI16BE
func ReadI16BE(r &: compress.ByteReader): Throws<int>The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI16LE
func ReadI16LE(r &: compress.ByteReader): Throws<int>The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI32BE
func ReadI32BE(r &: compress.ByteReader): Throws<int>The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI32LE
func ReadI32LE(r &: compress.ByteReader): Throws<int>The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI64BE
func ReadI64BE(r &: compress.ByteReader): Throws<i64>The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadI64LE
func ReadI64LE(r &: compress.ByteReader): Throws<i64>The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadF64
func ReadF64(r &: compress.ByteReader, big_endian: bool): Throws<f64>big_endian picks the byte order.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam
ReadF32
func ReadF32(r &: compress.ByteReader, big_endian: bool): Throws<f64>big_endian picks the byte order.The example for this item is shown with the ones above: encoding-binary-ReadWrite.salam