Standard library · Encoding and data

encoding.serde

Detecting the format of a piece of text and converting JSON, YAML, TOML, XML and CSV into one another.

import encoding.serde

This package converts text from one data format to another in one call. Transcode reads the input with the matching decoder, holds it as a Value, and writes it with the target encoder. Detect guesses which format a text is in. The format names are the lowercase strings json, yaml, toml, xml and csv; "JSON" is not recognised.

It suits tools that take whatever format a user hands them, and one-off migrations. When your program needs the data itself, call the format's own package (encoding.json, encoding.yaml, encoding.toml, encoding.xml, encoding.csv): they return a Value or fill a struct and give real error messages, while this package only answers yes or no. MessagePack is binary and is not one of the names here; use encoding.msgpack.

What survives a conversion

Everything passes through a Value, so comments and formatting are always lost, while keys keep the order they were read in. Beyond that, each format can hold less than JSON:

FormatReading givesWriting loses
JSON, YAMLevery kindYAML writes the float 2.0 as 2, which reads back as an int
TOMLevery kind except null; dates as stringsnull keys are dropped; the top level must be an object
XMLonly strings, objects and arrays (for repeated elements); attributes as @name, text as #text; the outer element's name is droppednumbers and booleans become text; the output is wrapped in <root>
CSVan array of objects, one per row, every cell a stringneeds an array of objects; only string fields are written, numbers and booleans come out as empty cells

What trips people up

  • Detection is a guess. The order is XML (text starts with <), JSON, TOML, CSV, then YAML, and nearly any plain text such as hello world is valid YAML. Pass the format yourself whenever a file extension or Content-Type tells you.
  • Transcode hides failures as an empty string. Use TranscodeStrict, which throws an Error that says what went wrong.
  • CSV in never fails, so any text read as CSV converts without an error.

Guide: accept any format

A tool that takes settings or data from users does not have to insist on one format. Detect looks at the text and names the format: XML when it starts with <, JSON when it parses as JSON, then TOML, CSV and finally YAML. Pass that name to TranscodeStrict to turn whatever came in into the one format your code reads.

encoding-serde-guide-anything.salam
import encoding.serde

func to_json(text: str): str:
    kind := serde.Detect(text)
    if kind == "":
        ret "(unknown format)"
    end
    try:
        ret kind + " -> " + serde.TranscodeStrict(text, kind, "json")
    catch e:
        ret "(could not read " + kind + ": " + e.message + ")"
    end
end

func main:
    println to_json("name: shop\nport: 8080\n")
    println to_json("name = \"shop\"\nport = 8080\n")
    println to_json("name,port\nshop,8080\n")
    println to_json(`{"name": "shop", "port": 8080}`)
    println to_json("")
end
Outputyaml -> {"name": "shop", "port": 8080} toml -> {"name": "shop", "port": 8080} csv -> [{"name": "shop", "port": "8080"}] json -> {"name": "shop", "port": 8080} (unknown format)

Detection is a guess from the shape of the text. Very short inputs can fit more than one format, and almost any plain text is valid YAML, so when you know the format, from a file extension or a Content-Type header, pass it yourself.

Guide: move a config to another format

Changing a project from YAML to TOML, or the other way, is one call. Convert, then check the result: read both the old and the new text back as JSON, decode them, and compare the two values with value.Equal. Comparing the JSON text instead would fail on key order alone.

encoding-serde-guide-migrate.salam
import encoding.serde
import encoding.json
import encoding.value

func main throws:
    old := `server:
  host: 0.0.0.0
  port: 8080
log:
  level: info
`
    as_toml := try serde.TranscodeStrict(old, "yaml", "toml")
    print as_toml

    before := try json.Decode(try serde.TranscodeStrict(old, "yaml", "json"))
    after := try json.Decode(try serde.TranscodeStrict(as_toml, "toml", "json"))
    println "same data:", value.Equal(before, after)
end
Output[server] host = "0.0.0.0" port = 8080 [log] level = "info" same data: true

Comments do not survive, because a Value holds data only, so tell people who edit the file by hand before you convert it.

Guide: XML in and out

XML has no arrays or numbers, so the mapping follows a few rules: an @name key becomes an attribute, an array becomes the same element repeated, #text becomes the element's text, and a converted document is wrapped in a <root> element. Reading XML gives every value as a string and attributes as @ keys.

encoding-serde-guide-xml.salam
import encoding.serde

func main:
    order := `{"@id": 1042, "customer": "Sara", "item": ["tea", "cups"], "total": 39.5, "paid": true}`
    x := serde.Transcode(order, "json", "xml")
    println x
    println "detected:", serde.Detect(x)
    println "back:", serde.Transcode(x, "xml", "json")
    println serde.Transcode("<book lang=\"fa\"><title>Salam</title><pages>153</pages></book>", "xml", "json")
end
Output<?xml version="1.0" encoding="UTF-8"?> <root id="1042"><customer>Sara</customer><item>tea</item><item>cups</item><total>39.5</total><paid>true</paid></root> detected: xml back: {"@id": "1042", "customer": "Sara", "item": ["tea", "cups"], "total": "39.5", "paid": "true"} {"@lang": "fa", "title": "Salam", "pages": "153"}

Numbers and booleans are written as their text, so 39.5 and true arrive in the XML. The way back shows what XML cannot express: every value returns as a string, and the outer <root> or <book> is gone, leaving only its contents. When the element names, order or namespaces matter, build the document with encoding.xml directly.

Guide: handle failure

TranscodeStrict throws when either half fails: reading in the source format, or writing in the target. Broken input throws the reader's own error, with its line and column; a format name it does not know throws InvalidArgument; data the target cannot hold, such as a top level array for TOML, throws Unsupported. Transcode just returns an empty string in all those cases.

encoding-serde-guide-errors.salam
import encoding.serde

func attempt(label: str, data: str, from: str, dest: str):
    try:
        println label, "ok:", serde.TranscodeStrict(data, from, dest)
    catch e:
        println label, "failed:", e.kind_name(), "->", e.ToStr()
    end
end

func main:
    attempt("good yaml   ", "- 1\n- 2\n", "yaml", "json")
    attempt("broken json ", `{"a": 1`, "json", "yaml")
    attempt("unknown from", `a = 1`, "ini", "json")
    attempt("unknown dest", `{"a": 1}`, "json", "ini")
    attempt("toml needs a table", `[1, 2]`, "json", "toml")
    println "lenient gives [" + serde.Transcode(`{"a": 1`, "json", "yaml") + "]"
end
Outputgood yaml ok: [1, 2] broken json failed: unexpected end of input -> json: unexpected end of input (line 1, column 8) unknown from failed: invalid argument -> serde: unknown format 'ini' unknown dest failed: invalid argument -> serde: unknown format 'ini' toml needs a table failed: unsupported -> toml: the top level must be an object, since a TOML document is a table, not an array lenient gives []

Constants and variables

FormatUnknown

const FormatUnknown: i32 = 0
An unknown format.
encoding-serde-Transcode.salam
import encoding.serde

func main:
    data := `{"name": "salam", "port": 8080, "tags": ["a", "b"]}`
    println serde.Detect(data), serde.Detect("a: 1\nb: 2\n"), serde.Detect("<a/>"), serde.Detect("[x]\ny = 1\n"), serde.Detect("a,b\n1,2\n")
    println serde.DetectFormat(data) == serde.FormatJSON, serde.FormatUnknown, serde.FormatYAML, serde.FormatTOML, serde.FormatXML, serde.FormatCSV
    println serde.Transcode(data, "json", "yaml")
    println serde.Transcode(data, "json", "toml")
    println serde.Transcode("name,age\nAli,30\n", "csv", "json")
    try:
        _ := serde.TranscodeStrict(data, "json", "pdf")
    catch e:
        println e.kind_name(), "->", e.message
    end
end
Outputjson yaml xml toml csv true 0 2 3 4 5 name: salam port: 8080 tags: - a - b name = "salam" port = 8080 tags = ["a", "b"] [{"name": "Ali", "age": "30"}] invalid argument -> unknown format 'pdf'

Functions

DetectFormat

func DetectFormat(text: str): int
Guesses which format the text is in and returns one of the Format constants: XML if it starts with <, then JSON, TOML, CSV and YAML, whichever parses first.

The example for this item is shown with the ones above: encoding-serde-Transcode.salam

Detect

func Detect(text: str): str
The same, but it returns the format's short name, or an empty string when nothing fits.

The example for this item is shown with the ones above: encoding-serde-Transcode.salam

TranscodeStrict

func TranscodeStrict(data: str, from: str, dest: str): Throws<str>
Converts text from the from format to dest. Throws an Error when reading or writing fails or a format name is wrong.

The example for this item is shown with the ones above: encoding-serde-Transcode.salam

Transcode

func Transcode(data: str, from: str, dest: str): str
Converts text from one format to another; any failure gives an empty string.

The example for this item is shown with the ones above: encoding-serde-Transcode.salam