Standard library · Encoding and data

encoding.query

Reading and building the query string of a web address, such as q=hello&page=2.

import encoding.query

A query string is the key=value&key=value text after the ? in a web address, and the same format is the body of an HTML form posted as application/x-www-form-urlencoded. This package reads and writes that text: it undoes the percent escapes and + for spaces, and builds correctly escaped strings from your values.

Use it when you hold the query text alone: a form body, the raw_query of a request, a signed API call. When you have a whole address and want to change its parameters, encoding.url has SetParam, RemoveParam and KeepParams on top of this package. Inside a net.http handler, http.Query and http.Form read single values for you; take http.RawQuery and this package when you need repeated keys or want to rewrite the query.

Two ways to work

  • On the text. Get, GetAll and Has take the query string itself. Each call decodes the whole string again, which is fine for reading a handful of values. They never fail: a % that is not followed by two hex digits is kept as written, the way browsers read such a query, so Get("off=50%", "off") is 50%.
  • On a QueryMap. Decode turns the text into a map you can change with Set, Add, Remove and Retain, then write back with Encode. One key may have several values (tag=a&tag=b), so each key holds a list, and the map remembers the order in which keys first appeared. Decode and DecodeWith are strict and declared throws: a broken escape throws an Error of kind Syntax, or UnexpectedEnd when the text stops inside an escape, with its column.

What trips people up

  • There is no getter on a QueryMap. Read the text with Get, or read the map directly: m.data.has("tag") and m.data.get("tag").items.
  • Missing and empty look the same to Get. Both page= and no page at all return ""; ask Has when it matters. A bare flag counts as present with an empty value, and Encode writes it back as flag=.
  • Numbers are not checked. "12abc".to_int() is 12 and "abc".to_int() is 0, so bound every number you read before using it.
  • Repeated keys are grouped. a=1&b=2&a=3 encodes back as a=1&a=3&b=2. That is the same data, but if a signature or cache key covers the exact text, sign what you send, not what you received.
  • Keys are case sensitive, and a leading ? is skipped by Decode.

A search page reads a few parameters, gives the missing ones a default, and converts the numbers. Get already decodes the value, so green+tea arrives as green tea. GetAll returns every value of a repeated key, which is how checkbox filters arrive.

encoding-query-guide-search.salam
import encoding.query

func main:
    qs := "q=green+tea&brand=nik&brand=pars&min=20&page=3&limit=500&ref=100%"

    words := query.Get(qs, "q")
    mut page := 1
    if query.Has(qs, "page"):
        page = query.Get(qs, "page").ToInt()
    end
    mut limit := 20
    if query.Has(qs, "limit"):
        limit = query.Get(qs, "limit").ToInt()
    end
    if page < 1: page = 1 end
    if limit < 1 or limit > 100: limit = 100 end
    brands := query.GetAll(qs, "brand")

    println "search for:", words
    println "brands:", brands.Len(), brands
    println "min price:", query.Get(qs, "min").ToInt()
    println "rows", (page - 1) * limit + 1, "to", page * limit
    println "ref kept as sent:", query.Get(qs, "ref")
end
Outputsearch for: green tea brands: 2 ["nik", "pars"] min price: 20 rows 201 to 300 ref kept as sent: 100%

Get returns an empty string both for a missing key and for key=, so use Has when the difference matters, as the defaults for page and limit do here. The ref value has a bare %; Get keeps it as sent instead of failing the whole page. Never trust the numbers: the visitor asked for 500 rows, and the handler caps limit at 100 and keeps page at 1 or more, so a bad value cannot make it read the whole table or a negative offset.

Guide: next and previous page links

Pagination links must keep every filter the visitor chose and change only the page number. Decode the current query into a QueryMap, Set the page (it replaces all values of that key, or adds the key when it is new) and Encode it again.

encoding-query-guide-pages.salam
import encoding.query
import str

func page_link(current: str, page: int): str throws:
    mut m := try query.Decode(current)
    query.Set(m, "page", str.FromInt(page))
    ret "/products?" + query.Encode(m)
end

func main throws:
    current := "category=books&sort=price&tag=new&tag=sale&page=2"
    println "prev:", try page_link(current, 1)
    println "next:", try page_link(current, 3)
    println "first visit:", try page_link("category=books", 2)
end
Outputprev: /products?category=books&sort=price&tag=new&tag=sale&page=1 next: /products?category=books&sort=price&tag=new&tag=sale&page=3 first visit: /products?category=books&page=2

Set keeps an existing key in its place, so the links differ from the address the visitor came from only in the page number, and a cache sees one address per page instead of one per parameter order.

Guide: HTML form bodies

A form sent with method="post" has the body type application/x-www-form-urlencoded, which is the same format as a query string. Decode it with the same functions. Spaces arrive as + and everything else outside plain ASCII as %XX bytes, Persian text included.

encoding-query-guide-form.salam
import encoding.query

func main throws:
    body := "name=Sara+Ahmadi&city=%D8%B4%DB%8C%D8%B1%D8%A7%D8%B2&note=50%25+off%3F&agree=on"
    form := try query.Decode(body)
    println "fields:", query.Keys(form)
    println "name:", query.Get(body, "name")
    println "city:", query.Get(body, "city")
    println "note:", query.Get(body, "note")
    println "agreed:", query.Get(body, "agree") == "on"

    mut reply := query.NewQueryMap()
    query.Set(reply, "status", "saved ok")
    println "form style:", query.Encode(reply)
    println "URL style: ", query.EncodeWith(reply, query.QueryOptions { space_as_plus = false })
end
Outputfields: ["name", "city", "note", "agree"] name: Sara Ahmadi city: شیراز note: 50% off? agreed: true form style: status=saved+ok URL style: status=saved%20ok

Encode writes spaces as +, as forms do. Some APIs want %20 instead; EncodeWith with space_as_plus = false writes that.

Guide: keep only the parameters you know

Parameters you do not use still change the address, which splits a cache into many copies of the same page and lets anyone pass odd flags along. Retain keeps only a list of allowed keys; Remove drops one and says whether it was there.

encoding-query-guide-allow.salam
import encoding.query

func main throws:
    incoming := "q=lamp&page=2&utm_source=ad&debug=1&sort=new&callback=x"

    mut m := try query.Decode(incoming)
    mut allowed := Vector {} as Vector<str>
    allowed.Push("q")
    allowed.Push("page")
    allowed.Push("sort")
    query.Retain(m, allowed)
    println "kept:", query.Encode(m)

    println "removed sort:", query.Remove(m, "sort")
    println "removed again:", query.Remove(m, "sort")
    println "cache key:", query.Encode(m), "with", query.Size(m), "keys"
end
Outputkept: q=lamp&page=2&sort=new removed sort: true removed again: false cache key: q=lamp&page=2 with 2 keys

Text functions or a QueryMap

JobUse
Read one or two values onceGet, GetAll, Has on the text
Change a query and write it backDecode, then Set, Add, Remove, Encode
Filter untrusted parametersRetain
Change the query of a full URLurl.SetParam and the other functions of encoding.url

Types

QueryOptions

struct QueryOptions
The encoding options.
NameTypeDescription
space_as_plusbool = trueForm style, true by default: a space is written as + and a + is read as a space. When false, %20 is written and a + is read as a plus sign.
encoding-query-Query.salam
import encoding.query

func main throws:
    text := "q=hello&page=2&tag=a&tag=b&empty="
    println query.Get(text, "q"), query.Get(text, "page"), query.Has(text, "empty"), query.Has(text, "x")
    println query.GetAll(text, "tag")
    mut m := try query.Decode(text)
    println query.Size(m), query.Keys(m)
    query.Add(m, "tag", "c")
    query.Set(m, "page", "3")
    println query.Remove(m, "empty")
    println query.Encode(m)
    mut keep := Vector {} as Vector<str>
    keep.Push("q")
    query.Retain(m, keep)
    println query.Encode(m)
    opts := query.QueryOptions {}
    println query.EncodeWith(try query.DecodeWith("b=2&a=1", opts), opts)
    empty := query.NewQueryMap()
    println query.Size(empty)
    println query.Get("off=50%", "off")
    try:
        _ := query.Decode("off=50%")
    catch e:
        println e.ToStr()
    end
end
Outputhello 2 true false ["a", "b"] 4 ["q", "page", "tag", "empty"] true q=hello&page=3&tag=a&tag=b&tag=c q=hello b=2&a=1 0 50% query: incomplete percent escape (line 1, column 7)

QueryValues

struct QueryValues
The values of one key.
NameTypeDescription
itemsVectorThe values, in order.

The example for this item is shown with the ones above: encoding-query-Query.salam

QueryMap

struct QueryMap
A map from key to values that keeps the key order.
NameTypeDescription
dataHashMapThe values of each key; read it with data.has(key) and data.get(key).items.
orderVectorThe keys in the order they first appeared.

The example for this item is shown with the ones above: encoding-query-Query.salam

Functions

NewQueryMap

func NewQueryMap(): QueryMap
An empty map.

The example for this item is shown with the ones above: encoding-query-Query.salam

DecodeWith

func DecodeWith(qs: str, opts: QueryOptions): Throws<QueryMap>
Reads a query string with the given options. Throws on a broken % escape, like Decode.

The example for this item is shown with the ones above: encoding-query-Query.salam

Decode

func Decode(qs: str): Throws<QueryMap>
Reads a query string; percent encoding and + are undone, a leading ? and empty pairs are skipped, and a key without = gets an empty value. Throws Syntax when a % is not followed by two hex digits and UnexpectedEnd when the text ends inside an escape.

The example for this item is shown with the ones above: encoding-query-Query.salam

EncodeWith

func EncodeWith(values: QueryMap, opts: QueryOptions): str
Turns a map into a string with the given options.

The example for this item is shown with the ones above: encoding-query-Query.salam

Encode

func Encode(values: QueryMap): str
Turns a map into a query string, keys in first-seen order with all values of a key together. Everything except letters, digits and -_.~ is percent-encoded.

The example for this item is shown with the ones above: encoding-query-Query.salam

GetAll

func GetAll(qs: str, key: str): Vector<str>
Every value of a key in the string. Never fails: a broken % escape is kept as written.

The example for this item is shown with the ones above: encoding-query-Query.salam

Get

func Get(qs: str, key: str): str
The first value of a key in the string, or an empty string when it is absent. Never fails: a broken % escape is kept as written.

The example for this item is shown with the ones above: encoding-query-Query.salam

Has

func Has(qs: str, key: str): bool
True when the key is in the string, even with an empty value. A broken escape in a key is kept as written.

The example for this item is shown with the ones above: encoding-query-Query.salam

Size

func Size(m: QueryMap): int
The number of distinct keys, not values.

The example for this item is shown with the ones above: encoding-query-Query.salam

Add

func Add(values &: QueryMap, key: str, val: str)
Adds another value to a key, keeping the earlier ones.

The example for this item is shown with the ones above: encoding-query-Query.salam

Set

func Set(values &: QueryMap, key: str, val: str)
Replaces every value of a key with one value; an existing key keeps its position, a new one goes at the end.

The example for this item is shown with the ones above: encoding-query-Query.salam

Keys

func Keys(values: QueryMap): Vector<str>
The keys, in order.

The example for this item is shown with the ones above: encoding-query-Query.salam

Remove

func Remove(values &: QueryMap, key: str): bool
Removes a key and its values and says whether it was there.

The example for this item is shown with the ones above: encoding-query-Query.salam

Retain

func Retain(values &: QueryMap, keep: Vector<str>)
Keeps only the keys you name.

The example for this item is shown with the ones above: encoding-query-Query.salam