encoding.serde
Detecting the format of a piece of text and converting JSON, YAML, TOML, XML and CSV into one another.
import encoding.serde
This package converts text from one data format to another in one call. Transcode reads the input with the matching decoder, holds it as a Value, and writes it with the target encoder. Detect guesses which format a text is in. The format names are the lowercase strings json, yaml, toml, xml and csv; "JSON" is not recognised.
It suits tools that take whatever format a user hands them, and one-off migrations. When your program needs the data itself, call the format's own package (encoding.json, encoding.yaml, encoding.toml, encoding.xml, encoding.csv): they return a Value or fill a struct and give real error messages, while this package only answers yes or no. MessagePack is binary and is not one of the names here; use encoding.msgpack.
What survives a conversion
Everything passes through a Value, so comments and formatting are always lost, while keys keep the order they were read in. Beyond that, each format can hold less than JSON:
| Format | Reading gives | Writing loses |
|---|---|---|
| JSON, YAML | every kind | YAML writes the float 2.0 as 2, which reads back as an int |
| TOML | every kind except null; dates as strings | null keys are dropped; the top level must be an object |
| XML | only strings, objects and arrays (for repeated elements); attributes as @name, text as #text; the outer element's name is dropped | numbers and booleans become text; the output is wrapped in <root> |
| CSV | an array of objects, one per row, every cell a string | needs an array of objects; only string fields are written, numbers and booleans come out as empty cells |
What trips people up
- Detection is a guess. The order is XML (text starts with
<), JSON, TOML, CSV, then YAML, and nearly any plain text such ashello worldis valid YAML. Pass the format yourself whenever a file extension orContent-Typetells you. Transcodehides failures as an empty string. UseTranscodeStrict, which throws an Error that says what went wrong.- CSV in never fails, so any text read as CSV converts without an error.
Guide: accept any format
A tool that takes settings or data from users does not have to insist on one format. Detect looks at the text and names the format: XML when it starts with <, JSON when it parses as JSON, then TOML, CSV and finally YAML. Pass that name to TranscodeStrict to turn whatever came in into the one format your code reads.
import encoding.serde
func to_json(text: str): str:
kind := serde.Detect(text)
if kind == "":
ret "(unknown format)"
end
try:
ret kind + " -> " + serde.TranscodeStrict(text, kind, "json")
catch e:
ret "(could not read " + kind + ": " + e.message + ")"
end
end
func main:
println to_json("name: shop\nport: 8080\n")
println to_json("name = \"shop\"\nport = 8080\n")
println to_json("name,port\nshop,8080\n")
println to_json(`{"name": "shop", "port": 8080}`)
println to_json("")
endDetection is a guess from the shape of the text. Very short inputs can fit more than one format, and almost any plain text is valid YAML, so when you know the format, from a file extension or a Content-Type header, pass it yourself.
Guide: move a config to another format
Changing a project from YAML to TOML, or the other way, is one call. Convert, then check the result: read both the old and the new text back as JSON, decode them, and compare the two values with value.Equal. Comparing the JSON text instead would fail on key order alone.
import encoding.serde
import encoding.json
import encoding.value
func main throws:
old := `server:
host: 0.0.0.0
port: 8080
log:
level: info
`
as_toml := try serde.TranscodeStrict(old, "yaml", "toml")
print as_toml
before := try json.Decode(try serde.TranscodeStrict(old, "yaml", "json"))
after := try json.Decode(try serde.TranscodeStrict(as_toml, "toml", "json"))
println "same data:", value.Equal(before, after)
endComments do not survive, because a Value holds data only, so tell people who edit the file by hand before you convert it.
Guide: XML in and out
XML has no arrays or numbers, so the mapping follows a few rules: an @name key becomes an attribute, an array becomes the same element repeated, #text becomes the element's text, and a converted document is wrapped in a <root> element. Reading XML gives every value as a string and attributes as @ keys.
import encoding.serde
func main:
order := `{"@id": 1042, "customer": "Sara", "item": ["tea", "cups"], "total": 39.5, "paid": true}`
x := serde.Transcode(order, "json", "xml")
println x
println "detected:", serde.Detect(x)
println "back:", serde.Transcode(x, "xml", "json")
println serde.Transcode("<book lang=\"fa\"><title>Salam</title><pages>153</pages></book>", "xml", "json")
endNumbers and booleans are written as their text, so 39.5 and true arrive in the XML. The way back shows what XML cannot express: every value returns as a string, and the outer <root> or <book> is gone, leaving only its contents. When the element names, order or namespaces matter, build the document with encoding.xml directly.
Guide: handle failure
TranscodeStrict throws when either half fails: reading in the source format, or writing in the target. Broken input throws the reader's own error, with its line and column; a format name it does not know throws InvalidArgument; data the target cannot hold, such as a top level array for TOML, throws Unsupported. Transcode just returns an empty string in all those cases.
import encoding.serde
func attempt(label: str, data: str, from: str, dest: str):
try:
println label, "ok:", serde.TranscodeStrict(data, from, dest)
catch e:
println label, "failed:", e.kind_name(), "->", e.ToStr()
end
end
func main:
attempt("good yaml ", "- 1\n- 2\n", "yaml", "json")
attempt("broken json ", `{"a": 1`, "json", "yaml")
attempt("unknown from", `a = 1`, "ini", "json")
attempt("unknown dest", `{"a": 1}`, "json", "ini")
attempt("toml needs a table", `[1, 2]`, "json", "toml")
println "lenient gives [" + serde.Transcode(`{"a": 1`, "json", "yaml") + "]"
endConstants and variables
FormatUnknown
const FormatUnknown: i32 = 0import encoding.serde
func main:
data := `{"name": "salam", "port": 8080, "tags": ["a", "b"]}`
println serde.Detect(data), serde.Detect("a: 1\nb: 2\n"), serde.Detect("<a/>"), serde.Detect("[x]\ny = 1\n"), serde.Detect("a,b\n1,2\n")
println serde.DetectFormat(data) == serde.FormatJSON, serde.FormatUnknown, serde.FormatYAML, serde.FormatTOML, serde.FormatXML, serde.FormatCSV
println serde.Transcode(data, "json", "yaml")
println serde.Transcode(data, "json", "toml")
println serde.Transcode("name,age\nAli,30\n", "csv", "json")
try:
_ := serde.TranscodeStrict(data, "json", "pdf")
catch e:
println e.kind_name(), "->", e.message
end
endFormatJSON
const FormatJSON: i32 = 1The example for this item is shown with the ones above: encoding-serde-Transcode.salam
FormatYAML
const FormatYAML: i32 = 2The example for this item is shown with the ones above: encoding-serde-Transcode.salam
FormatTOML
const FormatTOML: i32 = 3The example for this item is shown with the ones above: encoding-serde-Transcode.salam
FormatXML
const FormatXML: i32 = 4The example for this item is shown with the ones above: encoding-serde-Transcode.salam
FormatCSV
const FormatCSV: i32 = 5The example for this item is shown with the ones above: encoding-serde-Transcode.salam
Functions
DetectFormat
func DetectFormat(text: str): intFormat constants: XML if it starts with <, then JSON, TOML, CSV and YAML, whichever parses first.The example for this item is shown with the ones above: encoding-serde-Transcode.salam
Detect
func Detect(text: str): strThe example for this item is shown with the ones above: encoding-serde-Transcode.salam
TranscodeStrict
func TranscodeStrict(data: str, from: str, dest: str): Throws<str>from format to dest. Throws an Error when reading or writing fails or a format name is wrong.The example for this item is shown with the ones above: encoding-serde-Transcode.salam
Transcode
func Transcode(data: str, from: str, dest: str): strThe example for this item is shown with the ones above: encoding-serde-Transcode.salam