Standard library · Encoding and data

encoding.msgpack

The compact binary MessagePack format for dynamic values.

import encoding.msgpack

MessagePack holds the same data as JSON in a shorter binary form that is quicker to read, which suits messages between services, cache entries and RPC. This package turns a Value into a compress.ByteBuf with Marshal and reads one back with Unmarshal (or UnmarshalBytes for raw memory). Both are declared throws: bad data throws an Error, so call them with try.

Pick it over encoding.json when both ends are programs and size or speed matters; keep JSON when people read the data or a browser consumes it. For a fixed binary layout of your own, such as a file header, write the fields directly with encoding.binary; for schemas shared across languages, encoding.protobuf.

How values are written

ValueMessagePack
null, boolnil, true/false
intthe smallest int format that fits, from one byte up to 8
floatalways float64 (9 bytes); float32 is read but never written
stringfixstr, str8, str16 or str32 by length
array, objectfixarray/array16/array32, fixmap/map16/map32 with string keys

What trips people up

  • bin and ext are supported. A bin value arrives as KindBytes and an ext value as KindExt, with BytesToBuf and ExtDataToBuf giving the raw bytes. A string still stops at the first zero byte, so keep binary data in the value, not in a str.
  • Map keys must be strings. A map with integer keys, as some other libraries write, is read with every key turned into "", so the entries overwrite each other.
  • Trailing bytes are ignored. Unmarshal reads one value and does not report what follows it or how many bytes it used, so you cannot read several messages from one buffer. Frame each message yourself, for example with a 4-byte length in front.
  • Integers above i64 (a uint64 over 9223372036854775807) wrap around to negative numbers.

Guide: how much smaller is it?

MessagePack drops the quotes, colons and commas of JSON, and stores numbers and lengths in as few bytes as it can. For records with short keys and small numbers it usually comes to a half or two thirds of compact JSON. Decode JSON into a Value once, and Marshal writes it as MessagePack.

encoding-msgpack-guide-size.salam
import encoding.msgpack
import encoding.json
import encoding.value
import compress

func main throws:
    text := `[{"id": 1, "name": "tea", "price": 120, "stock": true},
 {"id": 2, "name": "coffee", "price": 90, "stock": true},
 {"id": 3, "name": "cocoa", "price": 75, "stock": false}]`
    rows := try json.Decode(text)

    compact := json.EncodeDefault(rows)
    packed := msgpack.Marshal(rows)
    println "JSON bytes:       ", compact.Len()
    println "MessagePack bytes:", compress.BufLen(packed)

    back := try msgpack.Unmarshal(packed)
    println "same data:", value.Equal(rows, back)
end
OutputJSON bytes: 169 MessagePack bytes: 90 same data: true

The keys are still written in full in every record, so long key names cost as much as in JSON. If size matters more than that, compress the buffer as well with compress.

Guide: what the bytes look like

Each value starts with a type byte. Small integers from 0 to 127 are a single byte, larger ones get a marker such as cc (one byte follows) or ce (four bytes follow). Short strings, arrays and maps carry their length in the first byte: a2 is a string of 2 bytes, 92 an array of 2 items, 81 a map with one pair.

encoding-msgpack-guide-bytes.salam
import encoding.msgpack
import encoding.value
import compress

func hex(v: value.Value): str:
    digits := "0123456789abcdef"
    mut b := msgpack.Marshal(v)
    mut out := ""
    repeat compress.BufLen(b) in i:
        x := compress.BufGet(b, i)
        if i > 0:
            out += " "
        end
        out = out + digits.Substr(x / 16, 1) + digits.Substr(x % 16, 1)
    end
    compress.BufFree(b)
    ret out
end

func main:
    println "5       ", hex(value.ValueInt(5))
    println "-1      ", hex(value.ValueInt(-1))
    println "200     ", hex(value.ValueInt(200))
    println "70000   ", hex(value.ValueInt(70000))
    println "true    ", hex(value.ValueBool(true))
    println "null    ", hex(value.ValueNull())
    println "1.5     ", hex(value.ValueFloat(1.5))
    println "\"hi\"    ", hex(value.ValueString("hi"))

    mut arr := value.NewValueArray()
    value.ArrayPush(arr, value.ValueInt(1))
    value.ArrayPush(arr, value.ValueInt(2))
    println "[1, 2]  ", hex(arr)

    mut obj := value.NewValueObject()
    value.ObjectSet(obj, "a", value.ValueInt(1))
    println "{\"a\": 1}", hex(obj)
end
Output5 05 -1 ff 200 cc c8 70000 ce 00 01 11 70 true c3 null c0 1.5 cb 3f f8 00 00 00 00 00 00 "hi" a2 68 69 [1, 2] 92 01 02 {"a": 1} 81 a1 61 01

The program reads the buffer byte by byte with compress.BufGet. Turning the buffer into a str first would not work: 70000 contains a zero byte, and a str ends at the first zero. Keep MessagePack data in its ByteBuf, or use BufToBase64 when you need text.

Guide: structs in a cache

A cache such as Redis stores bytes, and MessagePack bytes are quicker to read back than JSON. For a struct, go through JSON on the way in and out: json.Marshal and json.Decode give a Value to pack, and json.Unmarshal fills the struct from the unpacked Value. Here a HashMap stands in for the cache.

encoding-msgpack-guide-cache.salam
import encoding.msgpack
import encoding.json
import compress

struct Session:
    pub user: str = ""
    pub role: str = ""
    pub visits: int = 0
end

func save(cache &: HashMap<str, compress.ByteBuf>, key: str, s: Session) throws:
    v := try json.Decode(json.Marshal(s))
    cache.Put(key, msgpack.Marshal(v))
end

func load(cache: HashMap<str, compress.ByteBuf>, key: str, out &: Session): bool:
    if not cache.Has(key):
        ret false
    end
    try:
        v := msgpack.Unmarshal(cache.Get(key))
        json.Unmarshal(json.EncodeDefault(v), out)
    catch:
        ret false
    end
    ret true
end

func main throws:
    mut cache := HashMap {} as HashMap<str, compress.ByteBuf>
    try save(cache, "session:9f2", Session { user = "sara", role = "admin", visits = 12 })

    mut s := Session {}
    if load(cache, "session:9f2", s):
        println "found", s.user, s.role, s.visits
    end
    println "missing key loads:", load(cache, "session:000", s)
end
Outputfound sara admin 12 missing key loads: false

For a real cache, send b.data and b.len to the client library as a binary value, and decode what comes back with UnmarshalBytes.

Guide: broken, cut and foreign buffers

Network reads can stop halfway, and other programs may use parts of MessagePack this package does not read. Both Unmarshal and UnmarshalBytes throw an error of kind UnexpectedEnd for a buffer that ends in the middle of a value, so you can tell "wait for more bytes" apart from data that is simply wrong.

encoding-msgpack-guide-errors.salam
import encoding.msgpack
import encoding.value
import compress

func main throws:
    mut v := value.NewValueObject()
    value.ObjectSet(v, "name", value.ValueString("salam"))
    b := msgpack.Marshal(v)

    whole := try msgpack.UnmarshalBytes(b.data, b.len)
    println "whole buffer:", try whole.field("name").str()

    try:
        _ := msgpack.UnmarshalBytes(b.data, b.len - 3)
    catch e:
        println "cut short:", e.is(ErrorKind.UnexpectedEnd), e.ToStr()
    end

    mut blob := compress.NewByteBuf()
    compress.BufPush(blob, 0xc1)
    try:
        _ := msgpack.Unmarshal(blob)
    catch e:
        println "unsupported tag:", e.kind_name(), "->", e.message
    end

    mut two := compress.BufClone(b)
    compress.BufPush(two, 0x2a)
    first := try msgpack.Unmarshal(two)
    println "extra byte after the value:", value.ObjectLen(first), "key read"
end
Outputwhole buffer: salam cut short: true msgpack: unexpected end of input (offset 7) unsupported tag: unsupported -> unsupported msgpack tag 193 extra byte after the value: 1 key read

The third case is a bin value, type byte c4, as written by libraries that keep bytes apart from strings; it decodes to KindBytes. The last case shows that one stray byte after a complete value is not noticed, which is why a stream of messages needs its own framing.

Types

MsgpackOptions

struct MsgpackOptions
The serialising options.
NameTypeDescription
use_str8bool = trueUse the str8 format for strings of 32 to 255 bytes; true by default. Set it false for very old readers that predate str8; those strings are then written as str16.
encoding-msgpack-Marshal.salam
import encoding.msgpack
import encoding.value
import encoding
import compress

func main throws:
    mut v := value.NewValueObject()
    value.ObjectSet(v, "id", value.ValueInt(7))
    value.ObjectSet(v, "ok", value.ValueBool(true))
    b := msgpack.Marshal(v)
    println compress.BufLen(b), compress.BufToBase64(b)
    back := try msgpack.Unmarshal(b)
    println value.Equal(v, back), try back.field("id").int()
    back2 := try msgpack.UnmarshalBytes(b.data, b.len)
    println value.Equal(v, back2)
    o := msgpack.MsgpackOptions {}
    println compress.BufLen(msgpack.MarshalWith(v, o))
    println encoding.HexEncode(compress.BufToStrUnsafe(msgpack.Marshal(value.ValueInt(5))))
end
Output9 gqJpZAeib2vD true 7 true 9 05

Functions

MarshalWith

func MarshalWith(v: value.Value, opts: MsgpackOptions): compress.ByteBuf
Turns a value into MessagePack bytes with the given options.

The example for this item is shown with the ones above: encoding-msgpack-Marshal.salam

Marshal

func Marshal(v: value.Value): compress.ByteBuf
Turns a value into MessagePack bytes. Free the buffer with compress.BufFree when you are done.

The example for this item is shown with the ones above: encoding-msgpack-Marshal.salam

UnmarshalBytes

func UnmarshalBytes(p: void*, n: int): Throws<value.Value>
Reads one MessagePack value from n bytes at p, for data that is not in a ByteBuf, such as a reply from a cache client. Bytes after the value are ignored. Throws like Unmarshal.

The example for this item is shown with the ones above: encoding-msgpack-Marshal.salam

Unmarshal

func Unmarshal(b: compress.ByteBuf): Throws<value.Value>
Turns a MessagePack buffer into a value. Throws UnexpectedEnd when the data is cut short. bin becomes KindBytes and ext becomes KindExt.

The example for this item is shown with the ones above: encoding-msgpack-Marshal.salam