encoding.msgpack
The compact binary MessagePack format for dynamic values.
import encoding.msgpack
MessagePack holds the same data as JSON in a shorter binary form that is quicker to read, which suits messages between services, cache entries and RPC. This package turns a Value into a compress.ByteBuf with Marshal and reads one back with Unmarshal (or UnmarshalBytes for raw memory). Both are declared throws: bad data throws an Error, so call them with try.
Pick it over encoding.json when both ends are programs and size or speed matters; keep JSON when people read the data or a browser consumes it. For a fixed binary layout of your own, such as a file header, write the fields directly with encoding.binary; for schemas shared across languages, encoding.protobuf.
How values are written
| Value | MessagePack |
|---|---|
| null, bool | nil, true/false |
| int | the smallest int format that fits, from one byte up to 8 |
| float | always float64 (9 bytes); float32 is read but never written |
| string | fixstr, str8, str16 or str32 by length |
| array, object | fixarray/array16/array32, fixmap/map16/map32 with string keys |
What trips people up
binandextare supported. Abinvalue arrives asKindBytesand anextvalue asKindExt, withBytesToBufandExtDataToBufgiving the raw bytes. A string still stops at the first zero byte, so keep binary data in the value, not in astr.- Map keys must be strings. A map with integer keys, as some other libraries write, is read with every key turned into
"", so the entries overwrite each other. - Trailing bytes are ignored.
Unmarshalreads one value and does not report what follows it or how many bytes it used, so you cannot read several messages from one buffer. Frame each message yourself, for example with a 4-byte length in front. - Integers above
i64(auint64over 9223372036854775807) wrap around to negative numbers.
Guide: how much smaller is it?
MessagePack drops the quotes, colons and commas of JSON, and stores numbers and lengths in as few bytes as it can. For records with short keys and small numbers it usually comes to a half or two thirds of compact JSON. Decode JSON into a Value once, and Marshal writes it as MessagePack.
import encoding.msgpack
import encoding.json
import encoding.value
import compress
func main throws:
text := `[{"id": 1, "name": "tea", "price": 120, "stock": true},
{"id": 2, "name": "coffee", "price": 90, "stock": true},
{"id": 3, "name": "cocoa", "price": 75, "stock": false}]`
rows := try json.Decode(text)
compact := json.EncodeDefault(rows)
packed := msgpack.Marshal(rows)
println "JSON bytes: ", compact.Len()
println "MessagePack bytes:", compress.BufLen(packed)
back := try msgpack.Unmarshal(packed)
println "same data:", value.Equal(rows, back)
endThe keys are still written in full in every record, so long key names cost as much as in JSON. If size matters more than that, compress the buffer as well with compress.
Guide: what the bytes look like
Each value starts with a type byte. Small integers from 0 to 127 are a single byte, larger ones get a marker such as cc (one byte follows) or ce (four bytes follow). Short strings, arrays and maps carry their length in the first byte: a2 is a string of 2 bytes, 92 an array of 2 items, 81 a map with one pair.
import encoding.msgpack
import encoding.value
import compress
func hex(v: value.Value): str:
digits := "0123456789abcdef"
mut b := msgpack.Marshal(v)
mut out := ""
repeat compress.BufLen(b) in i:
x := compress.BufGet(b, i)
if i > 0:
out += " "
end
out = out + digits.Substr(x / 16, 1) + digits.Substr(x % 16, 1)
end
compress.BufFree(b)
ret out
end
func main:
println "5 ", hex(value.ValueInt(5))
println "-1 ", hex(value.ValueInt(-1))
println "200 ", hex(value.ValueInt(200))
println "70000 ", hex(value.ValueInt(70000))
println "true ", hex(value.ValueBool(true))
println "null ", hex(value.ValueNull())
println "1.5 ", hex(value.ValueFloat(1.5))
println "\"hi\" ", hex(value.ValueString("hi"))
mut arr := value.NewValueArray()
value.ArrayPush(arr, value.ValueInt(1))
value.ArrayPush(arr, value.ValueInt(2))
println "[1, 2] ", hex(arr)
mut obj := value.NewValueObject()
value.ObjectSet(obj, "a", value.ValueInt(1))
println "{\"a\": 1}", hex(obj)
endThe program reads the buffer byte by byte with compress.BufGet. Turning the buffer into a str first would not work: 70000 contains a zero byte, and a str ends at the first zero. Keep MessagePack data in its ByteBuf, or use BufToBase64 when you need text.
Guide: structs in a cache
A cache such as Redis stores bytes, and MessagePack bytes are quicker to read back than JSON. For a struct, go through JSON on the way in and out: json.Marshal and json.Decode give a Value to pack, and json.Unmarshal fills the struct from the unpacked Value. Here a HashMap stands in for the cache.
import encoding.msgpack
import encoding.json
import compress
struct Session:
pub user: str = ""
pub role: str = ""
pub visits: int = 0
end
func save(cache &: HashMap<str, compress.ByteBuf>, key: str, s: Session) throws:
v := try json.Decode(json.Marshal(s))
cache.Put(key, msgpack.Marshal(v))
end
func load(cache: HashMap<str, compress.ByteBuf>, key: str, out &: Session): bool:
if not cache.Has(key):
ret false
end
try:
v := msgpack.Unmarshal(cache.Get(key))
json.Unmarshal(json.EncodeDefault(v), out)
catch:
ret false
end
ret true
end
func main throws:
mut cache := HashMap {} as HashMap<str, compress.ByteBuf>
try save(cache, "session:9f2", Session { user = "sara", role = "admin", visits = 12 })
mut s := Session {}
if load(cache, "session:9f2", s):
println "found", s.user, s.role, s.visits
end
println "missing key loads:", load(cache, "session:000", s)
endFor a real cache, send b.data and b.len to the client library as a binary value, and decode what comes back with UnmarshalBytes.
Guide: broken, cut and foreign buffers
Network reads can stop halfway, and other programs may use parts of MessagePack this package does not read. Both Unmarshal and UnmarshalBytes throw an error of kind UnexpectedEnd for a buffer that ends in the middle of a value, so you can tell "wait for more bytes" apart from data that is simply wrong.
import encoding.msgpack
import encoding.value
import compress
func main throws:
mut v := value.NewValueObject()
value.ObjectSet(v, "name", value.ValueString("salam"))
b := msgpack.Marshal(v)
whole := try msgpack.UnmarshalBytes(b.data, b.len)
println "whole buffer:", try whole.field("name").str()
try:
_ := msgpack.UnmarshalBytes(b.data, b.len - 3)
catch e:
println "cut short:", e.is(ErrorKind.UnexpectedEnd), e.ToStr()
end
mut blob := compress.NewByteBuf()
compress.BufPush(blob, 0xc1)
try:
_ := msgpack.Unmarshal(blob)
catch e:
println "unsupported tag:", e.kind_name(), "->", e.message
end
mut two := compress.BufClone(b)
compress.BufPush(two, 0x2a)
first := try msgpack.Unmarshal(two)
println "extra byte after the value:", value.ObjectLen(first), "key read"
endThe third case is a bin value, type byte c4, as written by libraries that keep bytes apart from strings; it decodes to KindBytes. The last case shows that one stray byte after a complete value is not noticed, which is why a stream of messages needs its own framing.
Types
MsgpackOptions
struct MsgpackOptions| Name | Type | Description |
|---|---|---|
use_str8 | bool = true | Use the str8 format for strings of 32 to 255 bytes; true by default. Set it false for very old readers that predate str8; those strings are then written as str16. |
import encoding.msgpack
import encoding.value
import encoding
import compress
func main throws:
mut v := value.NewValueObject()
value.ObjectSet(v, "id", value.ValueInt(7))
value.ObjectSet(v, "ok", value.ValueBool(true))
b := msgpack.Marshal(v)
println compress.BufLen(b), compress.BufToBase64(b)
back := try msgpack.Unmarshal(b)
println value.Equal(v, back), try back.field("id").int()
back2 := try msgpack.UnmarshalBytes(b.data, b.len)
println value.Equal(v, back2)
o := msgpack.MsgpackOptions {}
println compress.BufLen(msgpack.MarshalWith(v, o))
println encoding.HexEncode(compress.BufToStrUnsafe(msgpack.Marshal(value.ValueInt(5))))
endFunctions
MarshalWith
func MarshalWith(v: value.Value, opts: MsgpackOptions): compress.ByteBufThe example for this item is shown with the ones above: encoding-msgpack-Marshal.salam
Marshal
func Marshal(v: value.Value): compress.ByteBufcompress.BufFree when you are done.The example for this item is shown with the ones above: encoding-msgpack-Marshal.salam
UnmarshalBytes
func UnmarshalBytes(p: void*, n: int): Throws<value.Value>n bytes at p, for data that is not in a ByteBuf, such as a reply from a cache client. Bytes after the value are ignored. Throws like Unmarshal.The example for this item is shown with the ones above: encoding-msgpack-Marshal.salam
Unmarshal
func Unmarshal(b: compress.ByteBuf): Throws<value.Value>UnexpectedEnd when the data is cut short. bin becomes KindBytes and ext becomes KindExt.The example for this item is shown with the ones above: encoding-msgpack-Marshal.salam