Standard library · Text

arabic

Normalising Arabic-script text: Arabic yeh and kaf, vowel marks, tatweel, the zero width non-joiner and digits, for search and comparison.

import arabic

Arabic-script text does not arrive in one shape. One source writes «كتاب» with an Arabic kaf and another with a Persian one; one adds vowel marks, another a tatweel (ـــ), and some text, from a PDF for instance, carries the presentation forms of the letters. To a reader they are the same word; to a computer they are different strings, and the search fails.

The arabic package removes those differences. Normalize works from flags you choose, combined with +. For most purposes NormalizePersian or SearchKey is all you need.

This package works on letters. To change how numbers look, str has ToPersianDigits and ToEnglishDigits; for the general Unicode forms behind it see str.NFKC and unicode; for displaying mixed Persian and English text in the right order see bidi.

Things that catch people out:

  • FoldPresentation, which NormalizePersian and SearchKey both use, runs the whole string through str.NFKC. That also rewrites characters outside Arabic script: the ligature fi becomes fi, ² becomes 2 and ½ becomes 1⁄2. Usually that is what you want, but not for a formula.
  • A space and a zero width non-joiner are not the same to the key: SearchKey removes the non-joiner, so هوش‌مند matches هوشمند but not هوش مند.
  • SearchKey lowers only ASCII letters, so Café and CAFÉ still get different keys.

Build a SearchKey for every stored text once, build one for the query, and compare keys instead of the original strings. A query typed on an Arabic keyboard (كتاب with an Arabic kaf, ايران with an Arabic yeh) then finds the Persian titles, and مدرسة with a teh marbuta finds مدرسهٔ.

arabic-guide-search.salam
import arabic
import str

func main:
    titles := ["کتاب آموزش برنامه‌نویسی", "تاریخ ایران", "دستور زبان فارسی", "مدرسهٔ عشق"]
    mut keys := Vector {} as Vector<str>
    defer keys.Free()
    each t in titles:
        keys.Push(arabic.SearchKey(t))
    end
    each query in ["كتاب", "برنامه نویسی", "ايران", "مدرسة", "فارسي"]:
        q := arabic.SearchKey(query)
        mut hits := 0
        each i, k in keys:
            if str.Contains(k, q):
                println "[" + query + "]", "found", titles[i]
                hits++
            end
        end
        if hits == 0:
            println "[" + query + "]", "no match"
        end
    end
end
Output[كتاب] found کتاب آموزش برنامه‌نویسی [برنامه نویسی] no match [ايران] found تاریخ ایران [مدرسة] found مدرسهٔ عشق [فارسي] found دستور زبان فارسی

One query fails, and it shows the most common trap: برنامه نویسی was typed with a space, while the title has a zero width non-joiner. The key removes the non-joiner, so the title becomes one word and the query stays two. If your users mix the two, also remove spaces from both keys with str.Replace(key, " ", "") before comparing. Show the original title to the user, never the key.

Finding duplicate names

The same person typed by different operators can arrive with an Arabic yeh, with a tatweel, or with a hamza on the alef. Using the search key as the key of a HashMap groups those spellings together, so a sign up form or a data import can warn about duplicates.

arabic-guide-dedupe.salam
import arabic

func main:
    typed := ["علی رضایی", "علي رضايي", "عـلی رضایی", "مریم احمدی", "مريم أحمدي", "سارا کریمی"]
    mut seen := HashMap {} as HashMap<str, str>
    defer seen.Free()
    each name in typed:
        key := arabic.SearchKey(name)
        if seen.Has(key):
            println "duplicate:", name, "is", seen.Get(key)
        else:
            seen.Put(key, name)
        end
    end
    println seen.Size(), "different people"
end
Outputduplicate: علي رضايي is علی رضایی duplicate: عـلی رضایی is علی رضایی duplicate: مريم أحمدي is مریم احمدی 3 different people

The map stores the first spelling that was seen, which is the one to keep and display. EqualLoose does the same comparison for just two strings, when you do not need a map.

Cleaning text for display or for a field

Text for people to read and text for a program to compare need different treatment. NormalizePersian is for display: it fixes the letters and the presentation forms that PDFs produce and removes tatweel, but keeps vowel marks, the non-joiner and the digits, so the text still reads as written. For a field such as a user name, combine exactly the flags you want with +.

arabic-guide-clean.salam
import arabic

func main:
    pasted := "ﻛﺘﺎﺏ مـــی‌خواهم، كيفيت ۱۰۰٪"
    display := arabic.NormalizePersian(pasted)
    println "display:", display
    username_flags := arabic.FoldLetters + arabic.StripTatweel + arabic.FoldDigits + arabic.FoldPresentation
    println "username:", arabic.Normalize("علي_۱۳۷۰ـ", username_flags)
    println "kept marks:", arabic.Normalize("آموزشِ زبان", arabic.FoldLetters)
    println "no marks:  ", arabic.Normalize("آموزشِ زبان", arabic.StripHarakat)
    println "loose equal:", arabic.EqualLoose(display, "کتاب میخواهم، کیفیت 100٪")
end
Outputdisplay: کتاب می‌خواهم، کیفیت ۱۰۰٪ username: علی_1370 kept marks: آموزشِ زبان no marks: آموزش زبان loose equal: true
PurposeUse
Text that people will readNormalizePersian
Search, indexes and duplicate checksSearchKey, EqualLoose
A user name or an idNormalize with FoldLetters + StripTatweel + FoldDigits + FoldPresentation
Removing only the vowel marksNormalize with StripHarakat
Checking single charactersIsArabicScript, IsHaraka, IsTatweel

Keep the original text and store the key next to it, for example in its own database column, rather than overwriting the text with the key. The key throws information away on purpose and cannot be turned back.

Constants and variables

FoldLetters

const FoldLetters: i32 = 1
A flag: converts Arabic letters to their Persian counterparts: "ي", "ى", "ے" and "ۓ" to "ی", and "ك", "ڪ" and "ګ" to "ک".
arabic-Normalize.salam
import arabic

func main:
    text := "كتابٌ عربيّة ـــ ٣٤ ﻻ"
    println arabic.NormalizePersian(text)
    println arabic.Normalize(text, arabic.StripHarakat)
    println arabic.Normalize(text, arabic.FoldLetters + arabic.FoldDigits)
    println arabic.Normalize("می‌خواهم", arabic.StripZWNJ)
    println arabic.Normalize("أحمد إيران", arabic.FoldAlef)
    println arabic.Normalize("مدرسة", arabic.FoldTehMarbuta)
    println arabic.Normalize("ﻻ ﻣﺮﺣﺒﺎ", arabic.FoldPresentation)
    println arabic.Normalize("ســلام", arabic.StripTatweel)
end
Outputکتابٌ عربیّة ٣٤ لا كتاب عربية ـــ ٣٤ ﻻ کتابٌ عربیّة ـــ 34 ﻻ میخواهم احمد ايران مدرسه لا مرحبا سلام

StripHarakat

const StripHarakat: i32 = 2
A flag: removes the vowel marks, fatha, kasra, damma, tanwin, shadda, the hamza above and below, the superscript alef and the Quranic marks. The single letter "ۀ" becomes "ه", the same as "هٔ" written with a separate hamza.

The example for this item is shown with the ones above: arabic-Normalize.salam

StripTatweel

const StripTatweel: i32 = 4
A flag: removes the tatweel (ـ).

The example for this item is shown with the ones above: arabic-Normalize.salam

FoldDigits

const FoldDigits: i32 = 8
A flag: converts Persian and Arabic digits to English ones.

The example for this item is shown with the ones above: arabic-Normalize.salam

FoldAlef

const FoldAlef: i32 = 16
A flag: converts the forms of alef (أ, إ, آ) to plain "ا".

The example for this item is shown with the ones above: arabic-Normalize.salam

FoldTehMarbuta

const FoldTehMarbuta: i32 = 32
A flag: converts "ة" to "ه".

The example for this item is shown with the ones above: arabic-Normalize.salam

StripZWNJ

const StripZWNJ: i32 = 64
A flag: removes the zero width non-joiner, so that «می‌خواهم» and «میخواهم» become one string.

The example for this item is shown with the ones above: arabic-Normalize.salam

FoldPresentation

const FoldPresentation: i32 = 128
A flag: converts the presentation forms of letters, such as «ﻻ» and «ﻣ», back to the ordinary letters. It does this with str.NFKC on the whole string, so other compatibility characters, such as ligatures and superscript digits, are rewritten too.

The example for this item is shown with the ones above: arabic-Normalize.salam

Functions

IsArabicScript

func IsArabicScript(cp: int): bool
True when the character belongs to the Arabic script, which includes the Persian letters.
arabic-IsArabicScript.salam
import arabic

func main:
    println arabic.IsArabicScript(1588), arabic.IsArabicScript(65)
    println arabic.IsHaraka(1614), arabic.IsHaraka(1588)
    println arabic.IsTatweel(1600), arabic.IsTatweel(1588)
end
Outputtrue false true false true false

IsHaraka

func IsHaraka(cp: int): bool
True when the character is one of the vowel marks.

The example for this item is shown with the ones above: arabic-IsArabicScript.salam

IsTatweel

func IsTatweel(cp: int): bool
True when the character is a tatweel (ـ).

The example for this item is shown with the ones above: arabic-IsArabicScript.salam

Normalize

func Normalize(s: str, flags: int): str
Normalises the text with the flags you give. Any aspect whose flag you leave out is left untouched.

The example for this item is shown with the ones above: arabic-Normalize.salam

NormalizePersian

func NormalizePersian(s: str): str
The default normalisation for Persian text: it converts Arabic letters to Persian ones, removes tatweel and expands presentation forms, but keeps the vowel marks, the zero width non-joiner and the digits, so the text still reads as it was written.

The example for this item is shown with the ones above: arabic-Normalize.salam

SearchKey

func SearchKey(s: str): str
Builds a key for searching: it applies every flag and then lowers the ASCII letters, so that two spellings a reader sees as the same produce the same key. Do not display the key; it is only for comparing and indexing.
arabic-SearchKey.salam
import arabic

func main:
    println arabic.SearchKey("كتابٌ"), arabic.SearchKey("کتاب")
    println arabic.EqualLoose("مي‌خواهم", "میخواهم")
    println arabic.EqualLoose("سلام", "کلام")
end
Outputکتاب کتاب true false

EqualLoose

func EqualLoose(a: str, b: str): bool
Compares two strings after building a search key for each, so that «مي‌خواهم» and «میخواهم» count as equal.

The example for this item is shown with the ones above: arabic-SearchKey.salam