Standard library · Encoding and data

encoding

Turning text and data into Base64, hexadecimal and URL encoding.

import encoding

This package holds the common ways of writing bytes as plain text: Base64 for fitting binary data into JSON or email, hexadecimal for showing bytes, and URL encoding for putting text or special characters into a web address.

The larger formats, such as JSON, YAML and XML, have sub-packages of their own, listed in this same group.

Every function takes and returns str, except the two ...Bytes forms, which work on a pointer and a length for binary data. The encoders cannot fail. The decoders Base64Decode, Base64DecodeBytes, HexDecode and URLDecode are declared throws: on bad input they throw an Error whose kind is Syntax (a character that does not belong), UnexpectedEnd (the text stops in the middle of a group, a pair or an escape) or OutOfRange (the buffer is too small), with the byte offset and the line and column of the problem. Neighbouring packages: encoding.binary for numbers packed into bytes, crypto for hashes that you then show in hex, and jwt for tokens.

The things people get wrong:

  • Decoding binary into a str. A str ends at the first zero byte, so Base64Decode and HexDecode cut binary data short; use Base64DecodeBytes into a buffer.
  • Forgetting that decoding can fail. Every decoder call needs try: inside a try: block to handle the error, or in front of the call in a function that is itself throws to pass it on.
  • Expecting HexDecode to skip spaces or a 0x prefix. It reads exactly two hex digits per byte and nothing else; strip separators first.
  • URLEncode writes a space as %20, while URLDecode reads both %20 and + as a space. A % that is not followed by two hex digits, such as the one in 50%, is an error; send it as %25.

Guide: an HTTP Basic login header

HTTP Basic authentication sends user:password in Base64 after the word Basic. Base64 is not encryption, anyone can decode it, so this only belongs on HTTPS. Building the header is one call; reading it on the server side means decoding with Base64Decode and splitting at the first colon, since the password itself may contain more colons.

encoding-guide-basic-auth.salam
import encoding

func basic_header(user: str, password: str): str:
    ret "Basic " + encoding.Base64Encode(user + ":" + password)
end

func read_header(header: str, user &: str, password &: str): bool:
    if header.Find("Basic ") != 0:
        ret false
    end
    mut pair := ""
    try:
        pair = encoding.Base64Decode(header.Substr(6, header.Len() - 6))
    catch e:
        println "refused:", e.ToStr()
        ret false
    end
    colon := pair.Find(":")
    if colon < 0:
        ret false
    end
    user = pair.Substr(0, colon)
    password = pair.Substr(colon + 1, pair.Len() - colon - 1)
    ret true
end

func main:
    h := basic_header("sara", "p@ss:2026")
    println "Authorization:", h

    mut user := ""
    mut password := ""
    if read_header(h, user, password):
        println "user:", user
        println "password:", password
    end
    println "bad header accepted:", read_header("Basic !!not-base64!!", user, password)
end
OutputAuthorization: Basic c2FyYTpwQHNzOjIwMjY= user: sara password: p@ss:2026 refused: base64: invalid character '!' (line 1, column 1) bad header accepted: false

Base64Decode throws when the text is not valid Base64, so a garbled header is refused instead of being turned into a random user name.

Guide: what the Base64 decoder accepts

Base64Decode accepts the forms Base64 really comes in: padded or unpadded, the standard alphabet (+ and /) or the URL-safe one (- and _), and spaces, tabs and line breaks anywhere, because MIME and PEM files wrap Base64 at 76 or 64 characters. Anything else is an error, so a typo is never turned into damaged bytes.

encoding-guide-strict.salam
import encoding

func check(label: str, s: str):
    try:
        println label, "ok:", encoding.Base64Decode(s)
    catch e:
        println label, "rejected:", e.kind_name(), "at column", e.column
    end
end

func main:
    check("padded   ", "U2FsYW0=")
    check("unpadded ", "U2FsYW0")
    check("wrapped  ", "U2Fs\nYW0=")
    check("url-safe ", "Pz8_")
    check("persian  ", encoding.Base64Encode("سلام"))
    check("typo     ", "U2Fs*W0=")
    check("cut short", "U2FsY")
    check("too much =", "U2E===")
end
Outputpadded ok: Salam unpadded ok: Salam wrapped ok: Salam url-safe ok: ??? persian ok: سلام typo rejected: syntax error at column 5 cut short rejected: unexpected end of input at column 6 too much = rejected: syntax error at column 5

A character outside both alphabets is a Syntax error at its position; so is anything after the = padding, or more = than the last group needs. Text that stops one character into a group of four, or that starts padding but leaves out an =, is UnexpectedEnd. For binary data use Base64EncodeBytes and Base64DecodeBytes, which work on a pointer and a length, because a str stops at the first zero byte.

Guide: seeing the bytes with hex

Hexadecimal writes each byte as two digits, which makes it the easiest way to see what a string really holds. Persian letters take two bytes each in UTF-8, so len of a Persian word counts bytes, not letters. Hex is also the usual form for hashes, keys and colours.

encoding-guide-hex.salam
import encoding

func main:
    words := ["A", "é", "س", "سلام"]
    each w in words:
        println w, "is", w.Len(), "bytes:", encoding.HexEncode(w)
    end

    stored := "d8b3d984d8a7d985"
    try:
        println "decoded:", encoding.HexDecode(stored)
        println "upper case works too:", encoding.HexDecode("D8B3D984D8A7D985")
        println encoding.HexDecode("d8b3d")
    catch e:
        println "error:", e.ToStr()
    end
end
OutputA is 1 bytes: 41 é is 2 bytes: c3a9 س is 2 bytes: d8b3 سلام is 8 bytes: d8b3d984d8a7d985 decoded: سلام upper case works too: سلام error: hex: odd number of hex digits (line 1, column 6)

HexEncode writes lower case; HexDecode reads both cases, and throws on an odd number of digits or a character that is not a hex digit.

Guide: putting text into a URL

A query value cannot contain spaces, &, = or non-ASCII letters as they are. URLEncode escapes everything except letters, digits and - _ . ~, so the result is safe anywhere in a URL. URLDecode undoes it and also reads + as a space, the way HTML forms send it; a broken escape such as 50%off makes it throw.

encoding-guide-url.salam
import encoding

func link(base: str, q: str, city: str): str:
    ret base + "?q=" + encoding.URLEncode(q) + "&city=" + encoding.URLEncode(city)
end

func main:
    url := link("https://example.com/search", "coffee & cake", "تهران")
    println url

    try:
        part := "coffee%20%26%20cake"
        println "q is", encoding.URLDecode(part)
        println "plus means space:", encoding.URLDecode("tea+time")
        println "round trip:", encoding.URLDecode(encoding.URLEncode("a/b?c=d")) == "a/b?c=d"
        println encoding.URLDecode("50%off")
    catch e:
        println "error:", e.ToStr()
    end
end
Outputhttps://example.com/search?q=coffee%20%26%20cake&city=%D8%AA%D9%87%D8%B1%D8%A7%D9%86 q is coffee & cake plus means space: tea time round trip: true error: url: malformed percent escape (line 1, column 3)

Encode each value on its own and join them with &, as link does: encoding the whole URL at once would also escape the ? and & that hold it together. For whole query strings with repeated keys, encoding.query does the bookkeeping, and encoding.url parses and rebuilds complete URLs.

Which encoding for which job

JobUse
Binary data inside JSON, email or a headerBase64Encode, Base64EncodeBytes
Reading Base64 from a client or a fileBase64Decode, Base64DecodeBytes
Hashes, keys, debugging outputHexEncode, HexDecode
One value in a URL path or queryURLEncode, URLDecode

Functions

Base64Encode

func Base64Encode(data: str): str
Turns text into standard Base64, padded with =.
encoding-Base64.salam
import encoding
import mem

func main:
    e := encoding.Base64Encode("hello world")
    println e
    b := mem.Allocate(3 as u64)
    mem.Set(b, 255, 3 as u64)
    println encoding.Base64EncodeBytes(b, 3)
    out := mem.Allocate(8 as u64)
    try:
        println encoding.Base64Decode(e)
        println encoding.Base64Decode("YWJj"), encoding.Base64Decode("YQ"), encoding.Base64Decode("Pz8_")
        println encoding.Base64DecodeBytes("AAEC", out, 8)
        println encoding.Base64Decode("***")
    catch err:
        println err.kind_name() + ": " + err.ToStr()
    end
    try:
        _ := encoding.Base64DecodeBytes("AAECAwQF", out, 4)
    catch err:
        println err.ToStr()
    end
    mem.Free(b)
    mem.Free(out)
end
OutputaGVsbG8gd29ybGQ= //// hello world abc a ??? 3 syntax error: base64: invalid character '*' (line 1, column 1) base64: decoded data needs 6 bytes but the buffer holds 4

Base64EncodeBytes

func Base64EncodeBytes(data: void*, n: int): str
Turns n bytes of memory into Base64.

The example for this item is shown with the ones above: encoding-Base64.salam

Base64Decode

func Base64Decode(s: str): Throws<str>
Turns Base64 back into text. Both the standard (+/) and the URL-safe (-_) alphabet are read, padding is optional and whitespace is skipped. Throws Syntax for any other character, data after the = padding or too much padding, and UnexpectedEnd when the text stops one character into a group or misses an =; the error has the offset, line and column.

The example for this item is shown with the ones above: encoding-Base64.salam

HexEncode

func HexEncode(data: str): str
Writes each byte of the text as two lower case hexadecimal digits.
encoding-Hex.salam
import encoding

func main:
    h := encoding.HexEncode("Hi!")
    println encoding.HexVal("f"), encoding.HexVal("A"), encoding.HexVal("z")
    u := encoding.URLEncode("name=Ali & age=30")
    println u
    try:
        println h, encoding.HexDecode(h)
        println encoding.URLDecode(u)
        println encoding.HexDecode("4869z1")
    catch e:
        println e.ToStr()
    end
    try:
        println encoding.URLDecode("100%")
    catch e:
        println e.ToStr()
    end
end
Output15 10 -1 name%3DAli%20%26%20age%3D30 486921 Hi! name=Ali & age=30 hex: invalid character 'z' (line 1, column 5) url: incomplete percent escape (line 1, column 4)

HexVal

func HexVal(c: str): int
The value of the first character as a hexadecimal digit, or -1 when it is not one.

The example for this item is shown with the ones above: encoding-Hex.salam

HexDecode

func HexDecode(s: str): Throws<str>
Turns a hexadecimal string back into bytes, reading upper and lower case. Throws Syntax at the first character that is not a hex digit and UnexpectedEnd when the number of digits is odd.

The example for this item is shown with the ones above: encoding-Hex.salam

URLEncode

func URLEncode(s: str): str
Percent-encodes text for a web address, byte by byte with upper case hex; letters, digits and -_.~ are left as they are, and a space becomes %20.

The example for this item is shown with the ones above: encoding-Hex.salam

URLDecode

func URLDecode(s: str): Throws<str>
Undoes percent encoding and turns + into a space. Throws Syntax when a % is followed by something other than two hex digits, and UnexpectedEnd when the text ends inside an escape.

The example for this item is shown with the ones above: encoding-Hex.salam

Base64DecodeBytes

func Base64DecodeBytes(s: str, out: void*, cap: int): Throws<int>
Decodes Base64, read as Base64Decode reads it, into the buffer out of capacity cap, and returns how many bytes it wrote. Throws the same errors as Base64Decode, and OutOfRange when the data needs more than cap bytes.

The example for this item is shown with the ones above: encoding-Base64.salam