arabic
Normalising Arabic-script text: Arabic yeh and kaf, vowel marks, tatweel, the zero width non-joiner and digits, for search and comparison.
import arabic
Arabic-script text does not arrive in one shape. One source writes «كتاب» with an Arabic kaf and another with a Persian one; one adds vowel marks, another a tatweel (ـــ), and some text, from a PDF for instance, carries the presentation forms of the letters. To a reader they are the same word; to a computer they are different strings, and the search fails.
The arabic package removes those differences. Normalize works from flags you choose, combined with +. For most purposes NormalizePersian or SearchKey is all you need.
This package works on letters. To change how numbers look, str has ToPersianDigits and ToEnglishDigits; for the general Unicode forms behind it see str.NFKC and unicode; for displaying mixed Persian and English text in the right order see bidi.
Things that catch people out:
FoldPresentation, whichNormalizePersianandSearchKeyboth use, runs the whole string throughstr.NFKC. That also rewrites characters outside Arabic script: the ligaturefibecomesfi,²becomes2and½becomes1⁄2. Usually that is what you want, but not for a formula.- A space and a zero width non-joiner are not the same to the key:
SearchKeyremoves the non-joiner, soهوشمندmatchesهوشمندbut notهوش مند. SearchKeylowers only ASCII letters, soCaféandCAFÉstill get different keys.
Searching titles whatever the spelling
Build a SearchKey for every stored text once, build one for the query, and compare keys instead of the original strings. A query typed on an Arabic keyboard (كتاب with an Arabic kaf, ايران with an Arabic yeh) then finds the Persian titles, and مدرسة with a teh marbuta finds مدرسهٔ.
import arabic
import str
func main:
titles := ["کتاب آموزش برنامهنویسی", "تاریخ ایران", "دستور زبان فارسی", "مدرسهٔ عشق"]
mut keys := Vector {} as Vector<str>
defer keys.Free()
each t in titles:
keys.Push(arabic.SearchKey(t))
end
each query in ["كتاب", "برنامه نویسی", "ايران", "مدرسة", "فارسي"]:
q := arabic.SearchKey(query)
mut hits := 0
each i, k in keys:
if str.Contains(k, q):
println "[" + query + "]", "found", titles[i]
hits++
end
end
if hits == 0:
println "[" + query + "]", "no match"
end
end
endOne query fails, and it shows the most common trap: برنامه نویسی was typed with a space, while the title has a zero width non-joiner. The key removes the non-joiner, so the title becomes one word and the query stays two. If your users mix the two, also remove spaces from both keys with str.Replace(key, " ", "") before comparing. Show the original title to the user, never the key.
Finding duplicate names
The same person typed by different operators can arrive with an Arabic yeh, with a tatweel, or with a hamza on the alef. Using the search key as the key of a HashMap groups those spellings together, so a sign up form or a data import can warn about duplicates.
import arabic
func main:
typed := ["علی رضایی", "علي رضايي", "عـلی رضایی", "مریم احمدی", "مريم أحمدي", "سارا کریمی"]
mut seen := HashMap {} as HashMap<str, str>
defer seen.Free()
each name in typed:
key := arabic.SearchKey(name)
if seen.Has(key):
println "duplicate:", name, "is", seen.Get(key)
else:
seen.Put(key, name)
end
end
println seen.Size(), "different people"
endThe map stores the first spelling that was seen, which is the one to keep and display. EqualLoose does the same comparison for just two strings, when you do not need a map.
Cleaning text for display or for a field
Text for people to read and text for a program to compare need different treatment. NormalizePersian is for display: it fixes the letters and the presentation forms that PDFs produce and removes tatweel, but keeps vowel marks, the non-joiner and the digits, so the text still reads as written. For a field such as a user name, combine exactly the flags you want with +.
import arabic
func main:
pasted := "ﻛﺘﺎﺏ مـــیخواهم، كيفيت ۱۰۰٪"
display := arabic.NormalizePersian(pasted)
println "display:", display
username_flags := arabic.FoldLetters + arabic.StripTatweel + arabic.FoldDigits + arabic.FoldPresentation
println "username:", arabic.Normalize("علي_۱۳۷۰ـ", username_flags)
println "kept marks:", arabic.Normalize("آموزشِ زبان", arabic.FoldLetters)
println "no marks: ", arabic.Normalize("آموزشِ زبان", arabic.StripHarakat)
println "loose equal:", arabic.EqualLoose(display, "کتاب میخواهم، کیفیت 100٪")
end| Purpose | Use |
|---|---|
| Text that people will read | NormalizePersian |
| Search, indexes and duplicate checks | SearchKey, EqualLoose |
| A user name or an id | Normalize with FoldLetters + StripTatweel + FoldDigits + FoldPresentation |
| Removing only the vowel marks | Normalize with StripHarakat |
| Checking single characters | IsArabicScript, IsHaraka, IsTatweel |
Keep the original text and store the key next to it, for example in its own database column, rather than overwriting the text with the key. The key throws information away on purpose and cannot be turned back.
Constants and variables
FoldLetters
const FoldLetters: i32 = 1import arabic
func main:
text := "كتابٌ عربيّة ـــ ٣٤ ﻻ"
println arabic.NormalizePersian(text)
println arabic.Normalize(text, arabic.StripHarakat)
println arabic.Normalize(text, arabic.FoldLetters + arabic.FoldDigits)
println arabic.Normalize("میخواهم", arabic.StripZWNJ)
println arabic.Normalize("أحمد إيران", arabic.FoldAlef)
println arabic.Normalize("مدرسة", arabic.FoldTehMarbuta)
println arabic.Normalize("ﻻ ﻣﺮﺣﺒﺎ", arabic.FoldPresentation)
println arabic.Normalize("ســلام", arabic.StripTatweel)
endStripHarakat
const StripHarakat: i32 = 2The example for this item is shown with the ones above: arabic-Normalize.salam
StripTatweel
const StripTatweel: i32 = 4The example for this item is shown with the ones above: arabic-Normalize.salam
FoldDigits
const FoldDigits: i32 = 8The example for this item is shown with the ones above: arabic-Normalize.salam
FoldAlef
const FoldAlef: i32 = 16The example for this item is shown with the ones above: arabic-Normalize.salam
FoldTehMarbuta
const FoldTehMarbuta: i32 = 32The example for this item is shown with the ones above: arabic-Normalize.salam
StripZWNJ
const StripZWNJ: i32 = 64The example for this item is shown with the ones above: arabic-Normalize.salam
FoldPresentation
const FoldPresentation: i32 = 128str.NFKC on the whole string, so other compatibility characters, such as ligatures and superscript digits, are rewritten too.The example for this item is shown with the ones above: arabic-Normalize.salam
Functions
IsArabicScript
func IsArabicScript(cp: int): boolimport arabic
func main:
println arabic.IsArabicScript(1588), arabic.IsArabicScript(65)
println arabic.IsHaraka(1614), arabic.IsHaraka(1588)
println arabic.IsTatweel(1600), arabic.IsTatweel(1588)
endIsHaraka
func IsHaraka(cp: int): boolThe example for this item is shown with the ones above: arabic-IsArabicScript.salam
IsTatweel
func IsTatweel(cp: int): boolThe example for this item is shown with the ones above: arabic-IsArabicScript.salam
Normalize
func Normalize(s: str, flags: int): strThe example for this item is shown with the ones above: arabic-Normalize.salam
NormalizePersian
func NormalizePersian(s: str): strThe example for this item is shown with the ones above: arabic-Normalize.salam
SearchKey
func SearchKey(s: str): strimport arabic
func main:
println arabic.SearchKey("كتابٌ"), arabic.SearchKey("کتاب")
println arabic.EqualLoose("ميخواهم", "میخواهم")
println arabic.EqualLoose("سلام", "کلام")
endEqualLoose
func EqualLoose(a: str, b: str): boolThe example for this item is shown with the ones above: arabic-SearchKey.salam