text
Converting a string to UTF-16 and back, for Windows and libraries that want that format.
import text
Salam strings are UTF-8, but some systems, Windows and a few C libraries above all, want text in UTF-16. In UTF-16 an ordinary character is one 16-bit "unit" and a rarer one, such as an emoji, takes two.
The text package converts in both directions and counts strings the way UTF-16 systems count them. Everything else you do with text, such as searching, splitting and changing case, lives in str; for character classes, normalisation and grapheme clusters see unicode.
Things to keep in mind:
- The memory
ToUtf16returns is yours. It ends with a zero unit and must be released withmem.Free. Utf16UnitAtdoes no bounds check. Stop at the zero unit; reading past it reads whatever memory follows.- The input is not validated. Broken UTF-8 is converted as best it can be, and a lone surrogate in a UTF-16 array is copied into the result as is, which gives a string that is not valid UTF-8.
Checking a length limit the way JavaScript counts
A browser, a JSON API written in JavaScript, Java or C#, and many databases measure a string in UTF-16 units. If your Salam server checks a limit in bytes or in characters, it can disagree with the form that sent the text: a limit of 12 that the browser enforced may look like 22 bytes to you, or a message the browser rejected may look short enough to Salam. Utf16Len counts the same way they do.
import text
import fmt
func fits(message: str, limit: int): str:
units := text.Utf16Len(message)
if units <= limit:
ret "fits"
end
ret "too long by " + (units - limit)
end
func main:
limit := 12
println fmt.PadRight("message", 16), "bytes chars units"
each m in ["Hello there!", "Salam donya :)", "Café au lait", "Party 🎉🎉🎉🎉"]:
println fmt.PadRight(m, 16), m.Len(), m.CharCount(), text.Utf16Len(m), fits(m, limit)
end
endThe three numbers per line are the byte length (len), the character count (char_count) and the UTF-16 length. For plain English they are equal. An accented letter adds a byte but not a unit. Each emoji is one character but two units, so the last message is only 10 characters long yet goes over a limit of 12.
Shortening text to a UTF-16 limit
When the other side counts in UTF-16, a preview, a push notification or an SMS has to be cut in the same units. Cutting by bytes can split a Persian letter, and cutting by units can split an emoji into half a surrogate pair, which shows up as a box or a question mark. Walking whole characters with str.Chars and adding up the units of each one avoids both.
import str
import text
func shorten(message: str, limit: int): str:
if text.Utf16Len(message) <= limit:
ret message
end
mut out := ""
mut used := 1
chars := str.Chars(message)
defer chars.Free()
each ch in chars:
units := text.Utf16Len(ch)
if used + units > limit:
break
end
out += ch
used += units
end
ret out + "…"
end
func main:
limit := 16
each m in ["Order shipped", "Your order #4471 has shipped", "Great news 🎉🎉🎉🎉🎉"]:
short := shorten(m, limit)
println short, "(" + text.Utf16Len(short) + " units)"
end
endOne unit is kept back for the … at the end, so the result never goes over the limit. The emoji line stops one unit short of 16 because the next emoji needs two units and only one is left. A sequence that the reader sees as one symbol, such as a flag or a family emoji, is made of several characters and can still be cut in the middle; for that, walk str.Graphemes instead of str.Chars.
Walking UTF-16 units and surrogate pairs
ToUtf16 builds a zero terminated array of 16-bit units, the shape that Windows functions with a W in their name and many C libraries want. Characters above U+FFFF, such as most emoji, take two units: a high surrogate between 55296 and 56319 followed by a low surrogate between 56320 and 57343. The example reads the units back one by one and joins each pair into its code point.
import text
import mem
func main:
s := "A€😀"
w := text.ToUtf16(s)
defer mem.Free(w)
n := text.Utf16Len(s)
mut i := 0
until i < n:
u := text.Utf16UnitAt(w, i)
if u >= 55296 and u < 56320:
lo := text.Utf16UnitAt(w, i + 1)
cp := 65536 + (u - 55296) * 1024 + (lo - 56320)
println "units", i, "and", i + 1, "form a pair: code point", cp
i += 2
else:
println "unit", i, "is code point", u
i++
end
end
println text.FromUtf16(w) == s
endThe array is allocated on the heap and belongs to you: release it with mem.Free, here through defer. FromUtf16 goes the other way and turns the units, pairs included, back into an ordinary Salam string, which is how you read text that a UTF-16 API handed back.
| Question | Use |
|---|---|
| How many bytes does it take to store or send? | s.len() |
| How many characters does it have? | s.char_count() or str.CharCount |
| How many letters does the reader see? | str.GraphemeCount |
| How long is it in JavaScript, Java or Windows? | text.Utf16Len |
Functions
Utf16Len
func Utf16Len(s: str): intimport text
func main:
s := "salam 😀"
println s.Len(), s.CharCount(), text.Utf16Len(s)
u := text.ToUtf16(s)
println text.Utf16UnitAt(u, 0), text.Utf16UnitAt(u, 6)
println text.FromUtf16(u)
endToUtf16
func ToUtf16(s: str): void*mem.Free.The example for this item is shown with the ones above: text-Utf16Len.salam
Utf16UnitAt
func Utf16UnitAt(w: void*, i: int): inti from a UTF-16 array, as a number from 0 to 65535. For an emoji you see two consecutive units, a surrogate pair. There is no bounds check, so stop at the zero unit.The example for this item is shown with the ones above: text-Utf16Len.salam
FromUtf16
func FromUtf16(w: void*): strnull pointer gives an empty string.The example for this item is shown with the ones above: text-Utf16Len.salam