Standard library · Basics

text

Converting a string to UTF-16 and back, for Windows and libraries that want that format.

import text

Salam strings are UTF-8, but some systems, Windows and a few C libraries above all, want text in UTF-16. In UTF-16 an ordinary character is one 16-bit "unit" and a rarer one, such as an emoji, takes two.

The text package converts in both directions and counts strings the way UTF-16 systems count them. Everything else you do with text, such as searching, splitting and changing case, lives in str; for character classes, normalisation and grapheme clusters see unicode.

Things to keep in mind:

  • The memory ToUtf16 returns is yours. It ends with a zero unit and must be released with mem.Free.
  • Utf16UnitAt does no bounds check. Stop at the zero unit; reading past it reads whatever memory follows.
  • The input is not validated. Broken UTF-8 is converted as best it can be, and a lone surrogate in a UTF-16 array is copied into the result as is, which gives a string that is not valid UTF-8.

Checking a length limit the way JavaScript counts

A browser, a JSON API written in JavaScript, Java or C#, and many databases measure a string in UTF-16 units. If your Salam server checks a limit in bytes or in characters, it can disagree with the form that sent the text: a limit of 12 that the browser enforced may look like 22 bytes to you, or a message the browser rejected may look short enough to Salam. Utf16Len counts the same way they do.

text-guide-limit.salam
import text
import fmt

func fits(message: str, limit: int): str:
    units := text.Utf16Len(message)
    if units <= limit:
        ret "fits"
    end
    ret "too long by " + (units - limit)
end

func main:
    limit := 12
    println fmt.PadRight("message", 16), "bytes chars units"
    each m in ["Hello there!", "Salam donya :)", "Café au lait", "Party 🎉🎉🎉🎉"]:
        println fmt.PadRight(m, 16), m.Len(), m.CharCount(), text.Utf16Len(m), fits(m, limit)
    end
end
Outputmessage bytes chars units Hello there! 12 12 12 fits Salam donya :) 14 14 14 too long by 2 Café au lait 13 12 12 fits Party 🎉🎉🎉🎉 22 10 14 too long by 2

The three numbers per line are the byte length (len), the character count (char_count) and the UTF-16 length. For plain English they are equal. An accented letter adds a byte but not a unit. Each emoji is one character but two units, so the last message is only 10 characters long yet goes over a limit of 12.

Shortening text to a UTF-16 limit

When the other side counts in UTF-16, a preview, a push notification or an SMS has to be cut in the same units. Cutting by bytes can split a Persian letter, and cutting by units can split an emoji into half a surrogate pair, which shows up as a box or a question mark. Walking whole characters with str.Chars and adding up the units of each one avoids both.

text-guide-truncate.salam
import str
import text

func shorten(message: str, limit: int): str:
    if text.Utf16Len(message) <= limit:
        ret message
    end
    mut out := ""
    mut used := 1
    chars := str.Chars(message)
    defer chars.Free()
    each ch in chars:
        units := text.Utf16Len(ch)
        if used + units > limit:
            break
        end
        out += ch
        used += units
    end
    ret out + "…"
end

func main:
    limit := 16
    each m in ["Order shipped", "Your order #4471 has shipped", "Great news 🎉🎉🎉🎉🎉"]:
        short := shorten(m, limit)
        println short, "(" + text.Utf16Len(short) + " units)"
    end
end
OutputOrder shipped (13 units) Your order #447… (16 units) Great news 🎉🎉… (16 units)

One unit is kept back for the … at the end, so the result never goes over the limit. The emoji line stops one unit short of 16 because the next emoji needs two units and only one is left. A sequence that the reader sees as one symbol, such as a flag or a family emoji, is made of several characters and can still be cut in the middle; for that, walk str.Graphemes instead of str.Chars.

Walking UTF-16 units and surrogate pairs

ToUtf16 builds a zero terminated array of 16-bit units, the shape that Windows functions with a W in their name and many C libraries want. Characters above U+FFFF, such as most emoji, take two units: a high surrogate between 55296 and 56319 followed by a low surrogate between 56320 and 57343. The example reads the units back one by one and joins each pair into its code point.

text-guide-pairs.salam
import text
import mem

func main:
    s := "A€😀"
    w := text.ToUtf16(s)
    defer mem.Free(w)
    n := text.Utf16Len(s)
    mut i := 0
    until i < n:
        u := text.Utf16UnitAt(w, i)
        if u >= 55296 and u < 56320:
            lo := text.Utf16UnitAt(w, i + 1)
            cp := 65536 + (u - 55296) * 1024 + (lo - 56320)
            println "units", i, "and", i + 1, "form a pair: code point", cp
            i += 2
        else:
            println "unit", i, "is code point", u
            i++
        end
    end
    println text.FromUtf16(w) == s
end
Outputunit 0 is code point 65 unit 1 is code point 8364 units 2 and 3 form a pair: code point 128512 true

The array is allocated on the heap and belongs to you: release it with mem.Free, here through defer. FromUtf16 goes the other way and turns the units, pairs included, back into an ordinary Salam string, which is how you read text that a UTF-16 API handed back.

QuestionUse
How many bytes does it take to store or send?s.len()
How many characters does it have?s.char_count() or str.CharCount
How many letters does the reader see?str.GraphemeCount
How long is it in JavaScript, Java or Windows?text.Utf16Len

Functions

Utf16Len

func Utf16Len(s: str): int
Returns the number of UTF-16 units in the string. For most scripts this equals the number of characters, but each emoji counts as two.
text-Utf16Len.salam
import text

func main:
    s := "salam 😀"
    println s.Len(), s.CharCount(), text.Utf16Len(s)
    u := text.ToUtf16(s)
    println text.Utf16UnitAt(u, 0), text.Utf16UnitAt(u, 6)
    println text.FromUtf16(u)
end
Output10 7 8 115 55357 salam 😀

ToUtf16

func ToUtf16(s: str): void*
Converts a string into a zero-terminated UTF-16 array and returns its address. The array is yours to release with mem.Free.

The example for this item is shown with the ones above: text-Utf16Len.salam

Utf16UnitAt

func Utf16UnitAt(w: void*, i: int): int
Reads unit number i from a UTF-16 array, as a number from 0 to 65535. For an emoji you see two consecutive units, a surrogate pair. There is no bounds check, so stop at the zero unit.

The example for this item is shown with the ones above: text-Utf16Len.salam

FromUtf16

func FromUtf16(w: void*): str
Converts a zero-terminated UTF-16 array back into a Salam string, joining surrogate pairs. A null pointer gives an empty string.

The example for this item is shown with the ones above: text-Utf16Len.salam