encoding
Turning text and data into Base64, hexadecimal and URL encoding.
import encoding
This package holds the common ways of writing bytes as plain text: Base64 for fitting binary data into JSON or email, hexadecimal for showing bytes, and URL encoding for putting text or special characters into a web address.
The larger formats, such as JSON, YAML and XML, have sub-packages of their own, listed in this same group.
Every function takes and returns str, except the two ...Bytes forms, which work on a pointer and a length for binary data. The encoders cannot fail. The decoders Base64Decode, Base64DecodeBytes, HexDecode and URLDecode are declared throws: on bad input they throw an Error whose kind is Syntax (a character that does not belong), UnexpectedEnd (the text stops in the middle of a group, a pair or an escape) or OutOfRange (the buffer is too small), with the byte offset and the line and column of the problem. Neighbouring packages: encoding.binary for numbers packed into bytes, crypto for hashes that you then show in hex, and jwt for tokens.
The things people get wrong:
- Decoding binary into a
str. Astrends at the first zero byte, soBase64DecodeandHexDecodecut binary data short; useBase64DecodeBytesinto a buffer. - Forgetting that decoding can fail. Every decoder call needs
try: inside atry:block to handle the error, or in front of the call in a function that is itselfthrowsto pass it on. - Expecting
HexDecodeto skip spaces or a0xprefix. It reads exactly two hex digits per byte and nothing else; strip separators first. URLEncodewrites a space as%20, whileURLDecodereads both%20and+as a space. A%that is not followed by two hex digits, such as the one in50%, is an error; send it as%25.
Guide: an HTTP Basic login header
HTTP Basic authentication sends user:password in Base64 after the word Basic. Base64 is not encryption, anyone can decode it, so this only belongs on HTTPS. Building the header is one call; reading it on the server side means decoding with Base64Decode and splitting at the first colon, since the password itself may contain more colons.
import encoding
func basic_header(user: str, password: str): str:
ret "Basic " + encoding.Base64Encode(user + ":" + password)
end
func read_header(header: str, user &: str, password &: str): bool:
if header.Find("Basic ") != 0:
ret false
end
mut pair := ""
try:
pair = encoding.Base64Decode(header.Substr(6, header.Len() - 6))
catch e:
println "refused:", e.ToStr()
ret false
end
colon := pair.Find(":")
if colon < 0:
ret false
end
user = pair.Substr(0, colon)
password = pair.Substr(colon + 1, pair.Len() - colon - 1)
ret true
end
func main:
h := basic_header("sara", "p@ss:2026")
println "Authorization:", h
mut user := ""
mut password := ""
if read_header(h, user, password):
println "user:", user
println "password:", password
end
println "bad header accepted:", read_header("Basic !!not-base64!!", user, password)
endBase64Decode throws when the text is not valid Base64, so a garbled header is refused instead of being turned into a random user name.
Guide: what the Base64 decoder accepts
Base64Decode accepts the forms Base64 really comes in: padded or unpadded, the standard alphabet (+ and /) or the URL-safe one (- and _), and spaces, tabs and line breaks anywhere, because MIME and PEM files wrap Base64 at 76 or 64 characters. Anything else is an error, so a typo is never turned into damaged bytes.
import encoding
func check(label: str, s: str):
try:
println label, "ok:", encoding.Base64Decode(s)
catch e:
println label, "rejected:", e.kind_name(), "at column", e.column
end
end
func main:
check("padded ", "U2FsYW0=")
check("unpadded ", "U2FsYW0")
check("wrapped ", "U2Fs\nYW0=")
check("url-safe ", "Pz8_")
check("persian ", encoding.Base64Encode("سلام"))
check("typo ", "U2Fs*W0=")
check("cut short", "U2FsY")
check("too much =", "U2E===")
endA character outside both alphabets is a Syntax error at its position; so is anything after the = padding, or more = than the last group needs. Text that stops one character into a group of four, or that starts padding but leaves out an =, is UnexpectedEnd. For binary data use Base64EncodeBytes and Base64DecodeBytes, which work on a pointer and a length, because a str stops at the first zero byte.
Guide: seeing the bytes with hex
Hexadecimal writes each byte as two digits, which makes it the easiest way to see what a string really holds. Persian letters take two bytes each in UTF-8, so len of a Persian word counts bytes, not letters. Hex is also the usual form for hashes, keys and colours.
import encoding
func main:
words := ["A", "é", "س", "سلام"]
each w in words:
println w, "is", w.Len(), "bytes:", encoding.HexEncode(w)
end
stored := "d8b3d984d8a7d985"
try:
println "decoded:", encoding.HexDecode(stored)
println "upper case works too:", encoding.HexDecode("D8B3D984D8A7D985")
println encoding.HexDecode("d8b3d")
catch e:
println "error:", e.ToStr()
end
endHexEncode writes lower case; HexDecode reads both cases, and throws on an odd number of digits or a character that is not a hex digit.
Guide: putting text into a URL
A query value cannot contain spaces, &, = or non-ASCII letters as they are. URLEncode escapes everything except letters, digits and - _ . ~, so the result is safe anywhere in a URL. URLDecode undoes it and also reads + as a space, the way HTML forms send it; a broken escape such as 50%off makes it throw.
import encoding
func link(base: str, q: str, city: str): str:
ret base + "?q=" + encoding.URLEncode(q) + "&city=" + encoding.URLEncode(city)
end
func main:
url := link("https://example.com/search", "coffee & cake", "تهران")
println url
try:
part := "coffee%20%26%20cake"
println "q is", encoding.URLDecode(part)
println "plus means space:", encoding.URLDecode("tea+time")
println "round trip:", encoding.URLDecode(encoding.URLEncode("a/b?c=d")) == "a/b?c=d"
println encoding.URLDecode("50%off")
catch e:
println "error:", e.ToStr()
end
endEncode each value on its own and join them with &, as link does: encoding the whole URL at once would also escape the ? and & that hold it together. For whole query strings with repeated keys, encoding.query does the bookkeeping, and encoding.url parses and rebuilds complete URLs.
Which encoding for which job
| Job | Use |
|---|---|
| Binary data inside JSON, email or a header | Base64Encode, Base64EncodeBytes |
| Reading Base64 from a client or a file | Base64Decode, Base64DecodeBytes |
| Hashes, keys, debugging output | HexEncode, HexDecode |
| One value in a URL path or query | URLEncode, URLDecode |
Functions
Base64Encode
func Base64Encode(data: str): str=.import encoding
import mem
func main:
e := encoding.Base64Encode("hello world")
println e
b := mem.Allocate(3 as u64)
mem.Set(b, 255, 3 as u64)
println encoding.Base64EncodeBytes(b, 3)
out := mem.Allocate(8 as u64)
try:
println encoding.Base64Decode(e)
println encoding.Base64Decode("YWJj"), encoding.Base64Decode("YQ"), encoding.Base64Decode("Pz8_")
println encoding.Base64DecodeBytes("AAEC", out, 8)
println encoding.Base64Decode("***")
catch err:
println err.kind_name() + ": " + err.ToStr()
end
try:
_ := encoding.Base64DecodeBytes("AAECAwQF", out, 4)
catch err:
println err.ToStr()
end
mem.Free(b)
mem.Free(out)
endBase64EncodeBytes
func Base64EncodeBytes(data: void*, n: int): strn bytes of memory into Base64.The example for this item is shown with the ones above: encoding-Base64.salam
Base64Decode
func Base64Decode(s: str): Throws<str>+/) and the URL-safe (-_) alphabet are read, padding is optional and whitespace is skipped. Throws Syntax for any other character, data after the = padding or too much padding, and UnexpectedEnd when the text stops one character into a group or misses an =; the error has the offset, line and column.The example for this item is shown with the ones above: encoding-Base64.salam
HexEncode
func HexEncode(data: str): strimport encoding
func main:
h := encoding.HexEncode("Hi!")
println encoding.HexVal("f"), encoding.HexVal("A"), encoding.HexVal("z")
u := encoding.URLEncode("name=Ali & age=30")
println u
try:
println h, encoding.HexDecode(h)
println encoding.URLDecode(u)
println encoding.HexDecode("4869z1")
catch e:
println e.ToStr()
end
try:
println encoding.URLDecode("100%")
catch e:
println e.ToStr()
end
endHexVal
func HexVal(c: str): intThe example for this item is shown with the ones above: encoding-Hex.salam
HexDecode
func HexDecode(s: str): Throws<str>Syntax at the first character that is not a hex digit and UnexpectedEnd when the number of digits is odd.The example for this item is shown with the ones above: encoding-Hex.salam
URLEncode
func URLEncode(s: str): str-_.~ are left as they are, and a space becomes %20.The example for this item is shown with the ones above: encoding-Hex.salam
URLDecode
func URLDecode(s: str): Throws<str>+ into a space. Throws Syntax when a % is followed by something other than two hex digits, and UnexpectedEnd when the text ends inside an escape.The example for this item is shown with the ones above: encoding-Hex.salam
Base64DecodeBytes
func Base64DecodeBytes(s: str, out: void*, cap: int): Throws<int>Base64Decode reads it, into the buffer out of capacity cap, and returns how many bytes it wrote. Throws the same errors as Base64Decode, and OutOfRange when the data needs more than cap bytes.The example for this item is shown with the ones above: encoding-Base64.salam